Nov 03 18:37:31.073956 ip-10-0-139-161 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:37:31.509278 ip-10-0-139-161 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:31.509278 ip-10-0-139-161 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:37:31.509278 ip-10-0-139-161 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:31.509278 ip-10-0-139-161 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:37:31.513628 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513547 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.513628 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513628 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513634 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513637 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513641 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513644 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513647 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513649 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513652 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513654 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513657 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513660 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513664 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513666 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513669 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513672 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513675 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513677 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513693 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513696 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.513709 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513699 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513701 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513704 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513707 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513710 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513713 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513715 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513718 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513721 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513724 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513726 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513729 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513732 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513735 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513737 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513740 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513742 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513745 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513747 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.514154 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513751 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513756 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513759 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513762 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513766 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513768 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513771 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513774 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513777 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513780 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513782 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513785 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513787 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513790 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513793 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513796 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513799 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513802 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513805 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513807 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.514602 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513810 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513812 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513818 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513820 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513823 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513825 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513828 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513830 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513833 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513835 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513838 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513841 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513844 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513846 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513849 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513852 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513854 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513856 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.515080 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513859 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513862 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513864 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513867 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513870 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513872 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513876 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513878 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513881 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513884 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513886 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513889 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513892 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513894 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513897 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513901 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513904 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513907 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513910 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.515523 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513912 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513915 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513918 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513922 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513924 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513928 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513931 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513933 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.513936 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514699 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514707 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514710 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514714 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514716 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514719 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514722 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514727 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514731 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514734 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.516015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514737 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514740 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514743 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514746 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514748 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514751 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514754 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514756 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514759 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514761 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514764 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514766 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514769 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514772 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514774 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514777 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514779 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514782 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514785 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514788 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.516479 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514790 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514793 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514795 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514798 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514800 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514803 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514805 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514808 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514811 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514813 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514816 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514818 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514822 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514825 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514828 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514830 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514833 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514835 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514838 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514840 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.516959 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514843 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514845 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514847 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514850 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514853 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514857 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514861 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514864 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514866 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514869 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514872 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514875 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514877 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514880 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514883 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514886 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514888 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514891 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514894 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514896 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.517455 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514899 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514902 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514904 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514907 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514910 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514913 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514915 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514918 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514920 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514922 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514925 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514928 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514930 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514933 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514935 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514938 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514940 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514943 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514945 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514948 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.518242 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514950 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514953 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514956 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514958 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514961 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514964 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514966 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514969 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514972 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514974 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514977 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514980 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514983 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514985 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514988 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.514990 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515445 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515450 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515453 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.518784 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515456 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515458 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515461 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515464 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515466 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515469 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515471 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515474 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515476 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515479 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515481 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515484 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515488 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515491 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515495 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515499 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515502 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515504 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515507 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.519260 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515510 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515512 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515515 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515517 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515519 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515522 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515524 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515527 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515530 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515533 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515537 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515539 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515542 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515544 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515547 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515549 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515552 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515554 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515557 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.519732 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515560 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515562 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515565 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515567 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515571 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515573 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515577 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515580 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515582 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515585 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515587 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515590 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515594 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515597 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515600 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515603 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515605 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515608 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515610 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.520175 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515613 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515615 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515618 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515621 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515625 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515627 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515630 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515632 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515635 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515637 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515640 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515642 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515644 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515647 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515650 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515652 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515655 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515657 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515660 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515662 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.520610 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515665 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515667 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515670 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515672 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515675 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515677 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515695 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515700 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515704 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515708 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515712 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515714 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515717 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515720 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515723 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515726 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515729 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515731 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515735 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515737 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.521126 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515740 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515743 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515745 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515748 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515750 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.515753 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516662 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516672 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516690 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516695 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516699 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516703 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516707 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516712 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516715 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516718 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516721 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516725 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516728 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516731 2579 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516734 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516737 2579 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516740 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:37:31.521668 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516743 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516747 2579 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516750 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516754 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516757 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516761 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516769 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516772 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516775 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516779 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516782 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516785 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516788 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516791 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516795 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516799 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516802 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516805 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516808 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516811 2579 flags.go:64] FLAG: --enable-server="true" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516814 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516819 2579 flags.go:64] FLAG: --event-burst="100" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516822 2579 flags.go:64] FLAG: --event-qps="50" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516825 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:37:31.522229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516829 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516832 2579 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516836 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516840 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516843 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516846 2579 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516849 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516852 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516854 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516857 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516860 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516863 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516866 2579 flags.go:64] FLAG: --feature-gates="" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516870 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516875 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516881 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516885 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516888 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516891 2579 flags.go:64] FLAG: --help="false" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516894 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516898 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516901 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516904 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:37:31.522857 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516907 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516910 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516913 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516916 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516919 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516922 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516924 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516927 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516930 2579 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516933 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516936 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516939 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516942 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516945 2579 flags.go:64] FLAG: --lock-file="" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516948 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516951 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516954 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516959 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516962 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516965 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516973 2579 flags.go:64] FLAG: --logging-format="text" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516975 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516979 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516983 2579 flags.go:64] FLAG: --manifest-url="" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516986 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:37:31.523414 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516991 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516995 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.516999 2579 flags.go:64] FLAG: --max-pods="110" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517002 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517005 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517009 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517012 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517015 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517018 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517021 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517029 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517032 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517035 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517038 2579 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517041 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517044 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517047 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517050 2579 flags.go:64] FLAG: --port="10250" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517053 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517056 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-028a19785a426fdba" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517060 2579 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517063 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517066 2579 flags.go:64] FLAG: --register-node="true" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517069 2579 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517073 2579 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517076 2579 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:37:31.524019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517079 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517083 2579 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517087 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517090 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517095 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517098 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517101 2579 flags.go:64] FLAG: --runonce="false" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517104 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517108 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517111 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517114 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517117 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517120 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517124 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517127 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517130 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517133 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517136 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517139 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517142 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517145 2579 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517148 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517153 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517156 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517159 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:37:31.524661 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517163 2579 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517166 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517169 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517172 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517175 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517178 2579 flags.go:64] FLAG: --v="2" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517182 2579 flags.go:64] FLAG: --version="false" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517186 2579 flags.go:64] FLAG: --vmodule="" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517191 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.517195 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519041 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519057 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519061 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519064 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519068 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519071 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519073 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519076 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519079 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519082 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519084 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519087 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.525303 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519090 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519093 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519095 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519098 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519100 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519103 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519105 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519108 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519111 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519114 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519117 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519119 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519122 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519125 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519127 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519130 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519133 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519135 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519138 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519141 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.525857 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519144 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519146 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519149 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519151 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519154 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519156 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519159 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519161 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519164 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519166 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519169 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519171 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519174 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519176 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519179 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519181 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519184 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519186 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519190 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519192 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.526336 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519195 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519197 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519200 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519203 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519206 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519211 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519213 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519216 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519218 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519221 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519224 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519227 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519229 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519232 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519234 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519237 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519239 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519242 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.526882 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519244 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519247 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519250 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519252 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519255 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519257 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519260 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519263 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519266 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519268 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519271 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519274 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519277 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519280 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519284 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519288 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519293 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519297 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519300 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.527334 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519303 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519306 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519309 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519312 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519315 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519318 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519321 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519323 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519326 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519328 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519331 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519334 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519336 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519339 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519341 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.527783 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519344 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.519373 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519419 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519423 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519426 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519429 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519432 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519435 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519437 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519440 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519442 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519445 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519448 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519451 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519454 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519457 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.528131 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519459 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519462 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519465 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519467 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519470 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519472 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519475 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519478 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519480 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519482 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519485 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519487 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519490 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519492 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519495 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519497 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519500 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519502 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519505 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519507 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.528520 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519510 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519512 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519514 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519517 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519519 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519522 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519524 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519527 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519530 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519532 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519536 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519538 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519541 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519543 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519546 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519548 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519550 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519553 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519556 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519558 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.529015 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519561 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519563 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519566 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519568 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519571 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519573 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519575 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519578 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519580 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519583 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519586 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519588 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519591 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519593 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519596 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519599 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519601 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519603 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519606 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.529484 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519609 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519611 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519614 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519617 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519621 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519624 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519626 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519629 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519631 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519634 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519636 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519639 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519644 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519647 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519651 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519654 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519657 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519660 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519663 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.529980 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519665 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519668 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519671 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519674 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519678 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519694 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519697 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519699 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519702 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519705 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519707 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519710 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519712 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.519715 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.526504 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.526518 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.527421 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.527430 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527487 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.530432 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527492 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527496 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527499 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527502 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527506 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527508 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527511 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527514 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527517 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527519 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527522 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527525 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527527 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527530 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527533 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527535 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527538 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527540 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.530898 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527544 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527548 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527553 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527556 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527559 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527562 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527565 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527568 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527570 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527573 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527577 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527580 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527582 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527585 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527587 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527590 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527592 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527595 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527598 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527600 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.531315 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527603 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527605 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527608 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527610 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527613 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527615 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527618 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527621 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527624 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527626 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527629 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527632 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527634 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527637 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527639 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527642 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527645 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527647 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527650 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527652 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.531817 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527655 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527657 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527660 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527663 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527666 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527668 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527671 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527673 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527676 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527678 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527695 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527698 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527700 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527703 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527706 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527709 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527712 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527714 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527716 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527719 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.532284 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527722 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527725 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527728 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527730 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527733 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527735 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527738 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527740 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527743 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527745 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527748 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527751 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527753 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527756 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527759 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527762 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527765 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527767 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527770 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.532868 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527772 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527775 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527777 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527779 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527782 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527785 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527787 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527790 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.527818 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527946 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527950 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527953 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527956 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527959 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527962 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527965 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.533302 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527967 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527970 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527972 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527975 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527977 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527980 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527983 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527985 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527988 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527990 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527993 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527996 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.527998 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528001 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528003 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528006 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528009 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528011 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528014 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.533706 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528016 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528019 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528021 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528024 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528026 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528029 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528031 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528034 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528036 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528041 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528043 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528046 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528049 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528052 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528054 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528057 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528060 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528062 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528065 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.534156 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528067 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528070 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528072 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528075 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528077 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528080 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528082 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528084 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528088 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528090 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528093 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528095 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528098 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528100 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528104 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528108 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528111 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528113 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528116 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528118 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.534653 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528121 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528123 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528126 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528128 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528131 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528133 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528136 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528139 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528142 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528144 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528147 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528149 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528152 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528154 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528157 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528159 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528162 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528164 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528167 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.535137 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528169 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528172 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528179 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528182 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528185 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528187 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528190 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528192 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528195 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528197 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528200 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528203 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528205 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528211 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528213 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528217 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528219 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528223 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528226 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.535582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528229 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528232 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.528255 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528286 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528290 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528294 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528297 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528300 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528303 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528305 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528308 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528310 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528313 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528316 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528318 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528321 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:37:31.536052 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528324 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528326 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528329 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528331 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528333 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528336 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528338 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528341 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528344 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528346 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528350 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528352 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528355 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528358 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528361 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528363 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528365 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528368 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528370 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528373 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:37:31.536419 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528375 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528378 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528380 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528383 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528386 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528388 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528390 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528393 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528395 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528398 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528400 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528403 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528405 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528408 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528411 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528414 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528416 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528419 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528421 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528424 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:37:31.536920 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528426 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528430 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528434 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528437 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528440 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528443 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528445 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528448 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528450 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528453 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528455 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528458 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528460 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528464 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528466 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528469 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528471 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528474 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528476 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:37:31.537384 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528479 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528481 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528484 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528486 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528489 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528491 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528494 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528496 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528499 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528501 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528504 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528506 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528509 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528511 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528514 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528518 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528521 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528524 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528526 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528529 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:37:31.537915 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528532 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528534 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528537 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528540 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528543 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528547 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528549 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528553 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528555 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528558 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528560 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528563 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528565 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:31.528568 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.528617 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.531529 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.532559 2579 server.go:1014] "Starting client certificate rotation" Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.532657 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:37:31.538378 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.533965 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:37:31.538845 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.534730 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:37:31.538845 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.535712 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:37:31.549981 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.549962 2579 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:37:31.553363 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.553350 2579 log.go:25] "Validated CRI v1 image API" Nov 03 18:37:31.554193 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.554166 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:37:31.557672 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.557655 2579 fs.go:135] Filesystem UUIDs: map[57f9321f-3745-4e7c-98f7-b24675b7da02:/dev/nvme0n1p4 63570359-2bb8-4093-82f0-7243a515b663:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 18:37:31.557727 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.557673 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 18:37:31.562867 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.562752 2579 manager.go:217] Machine: {Timestamp:2026-11-03 18:37:31.561194294 +0000 UTC m=+0.364230330 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089338 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c849cf7c1889eb667fb742e16b718 SystemUUID:ec2c849c-f7c1-889e-b667-fb742e16b718 BootID:418ba0f6-fd00-4fee-abc4-45df2f5d32e4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:dd:3e:19:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:dd:3e:19:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:68:df:95:a9:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:37:31.562867 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.562866 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:37:31.562981 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.562935 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:37:31.564844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.564821 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:37:31.564974 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.564850 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-161.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:37:31.565017 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.564976 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:37:31.565017 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.564984 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:37:31.565074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.565017 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:37:31.565074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.565034 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:37:31.566157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.566147 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:37:31.569307 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.569296 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:37:31.569415 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.569407 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:37:31.570343 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.570333 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:37:31.570381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.570345 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:37:31.570381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.570356 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:37:31.570381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.570364 2579 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:37:31.570381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.570377 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:37:31.570674 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.570663 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:37:31.572803 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.572785 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:37:31.578909 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.578891 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:37:31.579902 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.579888 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:37:31.579960 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.579919 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:37:31.580064 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.580054 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:37:31.580099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.580070 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:37:31.580099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.580077 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:37:31.580099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.580083 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:37:31.580099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.580089 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:37:31.580099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.580095 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:37:31.580999 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.580990 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:37:31.581033 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.581000 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:37:31.581033 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.581007 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:37:31.581033 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.581014 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:37:31.581033 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.581022 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:37:31.581033 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.581030 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:37:31.582456 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.582446 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:37:31.582456 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.582456 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:37:31.585987 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.585974 2579 server.go:1263] "Started kubelet" Nov 03 18:37:31.586100 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.586056 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:37:31.586174 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.586114 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:37:31.586174 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.586090 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:37:31.586666 ip-10-0-139-161 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:37:31.587511 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.587493 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:37:31.589085 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.589072 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:37:31.593041 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.593017 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 18:37:31.594853 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.594821 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:37:31.595380 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.595359 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:37:31.595619 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.595548 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" Nov 03 18:37:31.596110 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596092 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:37:31.596215 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596100 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:37:31.596215 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596131 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:37:31.596215 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596151 2579 factory.go:55] Registering systemd factory Nov 03 18:37:31.596215 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596165 2579 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:37:31.596371 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596220 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:37:31.596371 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596226 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:37:31.596422 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596405 2579 factory.go:153] Registering CRI-O factory Nov 03 18:37:31.596422 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596420 2579 factory.go:223] Registration of the crio container factory successfully Nov 03 18:37:31.596484 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596468 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:37:31.596530 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596487 2579 factory.go:103] Registering Raw factory Nov 03 18:37:31.596530 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.596500 2579 manager.go:1199] Started watching for new ooms in manager Nov 03 18:37:31.597662 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.597650 2579 manager.go:319] Starting recovery of all containers Nov 03 18:37:31.608022 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.608007 2579 manager.go:324] Recovery completed Nov 03 18:37:31.612944 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.612929 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:31.614600 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.614579 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:31.614702 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.614615 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:31.614702 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.614631 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:31.615143 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.615128 2579 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:37:31.615143 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.615142 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:31.615220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.615143 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:37:31.615220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.615209 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:37:31.615798 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.615782 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:31.615861 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.615809 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:31.615861 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.615824 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:31.616523 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.616511 2579 policy_none.go:50] "Start" Nov 03 18:37:31.616573 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.616529 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:37:31.616573 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.616539 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:37:31.696130 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.696107 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" Nov 03 18:37:31.739247 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.739203 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-161.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 18:37:31.739359 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.739231 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-161.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 18:37:31.742259 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.739158 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-161.us-west-2.compute.internal.18e49ce3485e4bd3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-161.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-161.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:31.585948627 +0000 UTC m=+0.388984657,LastTimestamp:2026-11-03 18:37:31.585948627 +0000 UTC m=+0.388984657,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-161.us-west-2.compute.internal,}" Nov 03 18:37:31.753506 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.753492 2579 policy_none.go:44] "Start" Nov 03 18:37:31.779487 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.779224 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:37:31.780387 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.780370 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:37:31.780387 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.780388 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:37:31.780513 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.780406 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:37:31.780513 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.780433 2579 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 18:37:31.786255 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.786229 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:37:31.788932 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.788818 2579 manager.go:341] "Starting Device Plugin manager" Nov 03 18:37:31.788999 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.788945 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:37:31.788999 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.788953 2579 server.go:87] "Starting device plugin registration server" Nov 03 18:37:31.789141 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.789132 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:37:31.789176 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.789143 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:37:31.789421 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.789264 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:37:31.789421 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.789358 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:37:31.789421 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.789368 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:37:31.789803 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.789785 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:37:31.789869 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.789824 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" Nov 03 18:37:31.823142 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.823121 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:37:31.823236 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.823181 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ht98s" Nov 03 18:37:31.880870 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.880849 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal"] Nov 03 18:37:31.880969 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.880915 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:31.881793 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.881778 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:31.881871 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.881806 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:31.881871 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.881816 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:31.883236 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.883224 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:31.883390 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.883374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.883448 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.883411 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:31.884456 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.884441 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:31.884456 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.884453 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:31.884548 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.884470 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:31.884548 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.884479 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:31.884548 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.884488 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:31.884548 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.884492 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:31.886041 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.886029 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.886104 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.886052 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:31.886822 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.886801 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:31.886921 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.886830 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:31.886921 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.886843 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:31.889695 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.889662 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:37:31.890488 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.890359 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:37:31.890488 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.890391 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:37:31.890488 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.890407 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:37:31.890488 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.890432 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.898856 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.898839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.898928 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.898863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.898928 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.898895 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a04c408b2b26c5648ca41d83d5e6490-config\") pod \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"3a04c408b2b26c5648ca41d83d5e6490\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.903334 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.903317 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" node="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.907267 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.907249 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" node="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.937278 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.937258 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:31.981740 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.981715 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:31.982251 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.982236 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:31.982848 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.982836 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:31.983292 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.983271 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.983378 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:31.983295 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-161.us-west-2.compute.internal\": node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" Nov 03 18:37:31.995931 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.995914 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.999147 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.999134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.999198 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.999156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.999198 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.999173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a04c408b2b26c5648ca41d83d5e6490-config\") pod \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"3a04c408b2b26c5648ca41d83d5e6490\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.999273 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.999209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a04c408b2b26c5648ca41d83d5e6490-config\") pod \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"3a04c408b2b26c5648ca41d83d5e6490\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.999273 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.999233 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:31.999273 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:31.999236 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:32.054036 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.053987 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:32.054799 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.054784 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:32.109848 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.109827 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:37:32.207534 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.207516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:32.211852 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.211517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:37:32.532699 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.532657 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 18:37:32.533195 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.532851 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:37:32.533195 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.532862 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:37:32.533195 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:32.532807 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://acc3b4ca8eba54ce98364924c8229c13-670cf6387112f978.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-161.us-west-2.compute.internal.18e49ce34a137392\": read tcp 10.0.139.161:41666->3.140.181.179:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-161.us-west-2.compute.internal.18e49ce34a137392 default 5605 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-161.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-161.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-161.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:37:31 +0000 UTC,LastTimestamp:2026-11-03 18:37:31.884454715 +0000 UTC m=+0.687490749,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-161.us-west-2.compute.internal,}" Nov 03 18:37:32.533195 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.532875 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 18:37:32.572655 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.572630 2579 apiserver.go:52] "Watching apiserver" Nov 03 18:37:32.595754 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.595735 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 18:37:32.828042 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.827958 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:32:32 +0000 UTC" deadline="2028-06-07 23:52:25.904848032 +0000 UTC" Nov 03 18:37:32.828042 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.827993 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13973h14m53.076859928s" Nov 03 18:37:32.887375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.887351 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:37:32.887658 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.887639 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs","openshift-dns/node-resolver-p2wnl","openshift-image-registry/node-ca-npkpz","openshift-multus/multus-8xwzz","openshift-network-diagnostics/network-check-target-gl7nv","openshift-network-operator/iptables-alerter-blnln","openshift-multus/multus-additional-cni-plugins-wlzkp","kube-system/global-pull-secret-syncer-2np47","kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-j9fq8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-df7m6","openshift-ovn-kubernetes/ovnkube-node-7gbhd","kube-system/konnectivity-agent-5sdxh"] Nov 03 18:37:32.889881 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.889854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.891946 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.891927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:32.893112 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.893096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:32.894537 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.894520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.895832 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.895818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:32.895902 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:32.895884 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:32.897052 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.897040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:32.898465 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.898446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:32.899850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.899835 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:32.899915 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:32.899897 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:32.901669 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.901657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:32.903193 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.903175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:32.903274 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:32.903241 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:32.904333 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14054e49-309c-46d4-ba51-5a2ad669f1a0-host\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:32.904424 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltb7\" (UniqueName: \"kubernetes.io/projected/14054e49-309c-46d4-ba51-5a2ad669f1a0-kube-api-access-vltb7\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:32.904424 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-socket-dir-parent\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.904424 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-daemon-config\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.904581 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.904581 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfq9n\" (UniqueName: \"kubernetes.io/projected/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kube-api-access-sfq9n\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.904581 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cnibin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.904581 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-hostroot\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.904814 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-device-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.904814 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-system-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.904814 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-netns\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.904814 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-socket-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.904814 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.904814 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-kubelet\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736cce59-1d70-4b20-975d-58d36563698f-iptables-alerter-script\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-hosts-file\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-tmp-dir\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-k8s-cni-cncf-io\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.904990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-bin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-conf-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfx8d\" (UniqueName: \"kubernetes.io/projected/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-kube-api-access-bfx8d\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-registration-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-sys-fs\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14054e49-309c-46d4-ba51-5a2ad669f1a0-serviceca\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:32.905197 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-etc-kubernetes\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.906074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtjx\" (UniqueName: \"kubernetes.io/projected/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-kube-api-access-mhtjx\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:32.906074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.906074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cni-binary-copy\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.906074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-multus-certs\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.906074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-multus\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:32.906074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736cce59-1d70-4b20-975d-58d36563698f-host-slash\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:32.906074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.905585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmnm\" (UniqueName: \"kubernetes.io/projected/736cce59-1d70-4b20-975d-58d36563698f-kube-api-access-2kmnm\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:32.907157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.907141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:32.949260 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.949237 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:37:32.949359 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.949327 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:37:32.949359 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.949328 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:37:32.949474 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.949435 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cxqpk\"" Nov 03 18:37:32.950385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.950351 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r88kn\"" Nov 03 18:37:32.950469 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.950447 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:37:32.950579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.950561 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:37:32.951512 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.951493 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:37:32.951512 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.951495 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fft52\"" Nov 03 18:37:32.951648 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.951527 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:37:32.951738 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.951725 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:37:32.952956 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.952943 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:37:32.953034 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.953021 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-btttq\"" Nov 03 18:37:32.997172 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:32.997154 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:37:33.006315 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.006293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmnm\" (UniqueName: \"kubernetes.io/projected/736cce59-1d70-4b20-975d-58d36563698f-kube-api-access-2kmnm\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:33.006397 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.006331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:33.006397 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.006377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06841536-c400-4c0f-8d46-5d8e24ba9c6d-konnectivity-ca\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:33.006479 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.006456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-kubernetes\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.006534 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.006501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-sys\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.007103 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.007207 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14054e49-309c-46d4-ba51-5a2ad669f1a0-host\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:33.007207 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-daemon-config\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.007207 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-slash\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.007207 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007187 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14054e49-309c-46d4-ba51-5a2ad669f1a0-host\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:33.007381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-etc-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.007381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-config\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.007381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-env-overrides\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.007381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06841536-c400-4c0f-8d46-5d8e24ba9c6d-agent-certs\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:33.007381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-lib-modules\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.007529 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.007529 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-hostroot\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.007529 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-etc-tuned\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.007529 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.007529 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-hostroot\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.007529 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fjt\" (UniqueName: \"kubernetes.io/projected/0f0cd7d8-5868-443d-88a4-8d2e464631e9-kube-api-access-h9fjt\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.007726 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-device-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.007726 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-systemd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.007726 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-run\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.007726 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-device-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.007726 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.007726 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736cce59-1d70-4b20-975d-58d36563698f-iptables-alerter-script\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:33.007933 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.007933 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-netd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.007933 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-modprobe-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.007933 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-var-lib-kubelet\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.007933 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmjb\" (UniqueName: \"kubernetes.io/projected/38dd707d-5338-4e99-a3b4-49380731aebb-kube-api-access-szmjb\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.007933 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-kubelet-config\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.007933 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-hosts-file\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-tmp-dir\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-k8s-cni-cncf-io\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.007983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-hosts-file\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008054 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-k8s-cni-cncf-io\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rfm\" (UniqueName: \"kubernetes.io/projected/4322a265-eccb-4e0b-89e3-8844e35f3b6e-kube-api-access-p5rfm\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008110 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-netns\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008165 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-tmp-dir\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-sys-fs\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14054e49-309c-46d4-ba51-5a2ad669f1a0-serviceca\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-etc-kubernetes\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-sys-fs\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.008286 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-conf\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008314 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-etc-kubernetes\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736cce59-1d70-4b20-975d-58d36563698f-host-slash\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-kubelet\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736cce59-1d70-4b20-975d-58d36563698f-host-slash\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltb7\" (UniqueName: \"kubernetes.io/projected/14054e49-309c-46d4-ba51-5a2ad669f1a0-kube-api-access-vltb7\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-socket-dir-parent\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008570 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14054e49-309c-46d4-ba51-5a2ad669f1a0-serviceca\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-script-lib\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysconfig\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfq9n\" (UniqueName: \"kubernetes.io/projected/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kube-api-access-sfq9n\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cnibin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-ovn\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-socket-dir-parent\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008829 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cnibin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.008850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-system-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008919 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-system-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-netns\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-var-lib-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktm4\" (UniqueName: \"kubernetes.io/projected/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-kube-api-access-5ktm4\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008988 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-netns\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.008993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-tmp\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-system-cni-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-socket-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-kubelet\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-kubelet\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-systemd\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cnibin\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009171 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-socket-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.009481 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-bin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-conf-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-bin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx8d\" (UniqueName: \"kubernetes.io/projected/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-kube-api-access-bfx8d\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-bin\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-conf-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovn-node-metrics-cert\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-registration-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-systemd-units\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009492 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-registration-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-host\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-dbus\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtjx\" (UniqueName: \"kubernetes.io/projected/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-kube-api-access-mhtjx\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010156 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cni-binary-copy\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-multus-certs\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-node-log\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.010652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-multus-certs\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-log-socket\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.010652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-multus\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.010652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.009878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-multus\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.054542 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.054518 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 18:37:33.054629 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.054572 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-527s5" Nov 03 18:37:33.058576 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058555 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:37:33.058646 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058598 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:37:33.058646 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058617 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:37:33.058850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058734 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:37:33.058850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058763 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:37:33.058850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058774 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:37:33.058850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058795 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:33.058850 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058811 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058952 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058963 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.058974 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxjph\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059057 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059096 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059139 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059193 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059203 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lzn26\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059202 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:37:33.059219 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059222 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d4mhq\"" Nov 03 18:37:33.059531 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.059231 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:37:33.060657 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.060642 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczj\"" Nov 03 18:37:33.060657 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.060653 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:37:33.060777 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.060742 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cni-binary-copy\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.068781 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.068748 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-daemon-config\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.068842 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.068787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736cce59-1d70-4b20-975d-58d36563698f-iptables-alerter-script\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:33.077646 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.077597 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zsb8x\"" Nov 03 18:37:33.078017 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.077993 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:33.078017 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.078020 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:33.078306 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.078034 2579 projected.go:196] Error preparing data for projected volume kube-api-access-c5zw8 for pod openshift-network-diagnostics/network-check-target-gl7nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:33.078306 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.078042 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:37:33.078306 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.078090 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8 podName:aedaa821-137b-4371-9a23-c4a2890cd484 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:33.578075324 +0000 UTC m=+2.381111360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c5zw8" (UniqueName: "kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8") pod "network-check-target-gl7nv" (UID: "aedaa821-137b-4371-9a23-c4a2890cd484") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:33.078306 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.078203 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:37:33.081909 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.081887 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx8d\" (UniqueName: \"kubernetes.io/projected/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-kube-api-access-bfx8d\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.082020 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.081931 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfq9n\" (UniqueName: \"kubernetes.io/projected/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kube-api-access-sfq9n\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.082020 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.081967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltb7\" (UniqueName: \"kubernetes.io/projected/14054e49-309c-46d4-ba51-5a2ad669f1a0-kube-api-access-vltb7\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:33.082096 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.082082 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmnm\" (UniqueName: \"kubernetes.io/projected/736cce59-1d70-4b20-975d-58d36563698f-kube-api-access-2kmnm\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:33.082137 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.082095 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtjx\" (UniqueName: \"kubernetes.io/projected/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-kube-api-access-mhtjx\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:33.111029 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111029 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-systemd\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.111181 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cnibin\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.111181 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-bin\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111181 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cnibin\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.111181 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111117 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-systemd\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.111181 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111181 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-bin\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111181 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovn-node-metrics-cert\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-systemd-units\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-host\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-dbus\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111330 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-systemd-units\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-node-log\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-log-socket\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111397 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-dbus\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-host\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06841536-c400-4c0f-8d46-5d8e24ba9c6d-konnectivity-ca\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:33.111466 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-kubernetes\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-node-log\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.111482 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-sys\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.111538 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs podName:4322a265-eccb-4e0b-89e3-8844e35f3b6e nodeName:}" failed. No retries permitted until 2026-11-03 18:37:33.611519976 +0000 UTC m=+2.414556024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs") pod "network-metrics-daemon-df7m6" (UID: "4322a265-eccb-4e0b-89e3-8844e35f3b6e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111549 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-log-socket\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-slash\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.111594 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111635 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-etc-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.111642 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret podName:4d25cdfb-758e-47fa-adbc-8f6953a7be99 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:33.611626403 +0000 UTC m=+2.414662422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret") pod "global-pull-secret-syncer-2np47" (UID: "4d25cdfb-758e-47fa-adbc-8f6953a7be99") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-etc-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111633 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-sys\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-config\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111682 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-slash\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-env-overrides\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06841536-c400-4c0f-8d46-5d8e24ba9c6d-agent-certs\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:33.112157 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-lib-modules\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-etc-tuned\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fjt\" (UniqueName: \"kubernetes.io/projected/0f0cd7d8-5868-443d-88a4-8d2e464631e9-kube-api-access-h9fjt\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-systemd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-run\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-netd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111898 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-kubernetes\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-modprobe-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-var-lib-kubelet\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmjb\" (UniqueName: \"kubernetes.io/projected/38dd707d-5338-4e99-a3b4-49380731aebb-kube-api-access-szmjb\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-kubelet-config\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.111987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rfm\" (UniqueName: \"kubernetes.io/projected/4322a265-eccb-4e0b-89e3-8844e35f3b6e-kube-api-access-p5rfm\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-netns\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-conf\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-kubelet\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113012 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-script-lib\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-modprobe-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysconfig\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-systemd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112242 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-var-lib-kubelet\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysconfig\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112337 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-lib-modules\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112353 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-conf\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112381 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-netd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-netns\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-run\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112481 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-kubelet-config\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112462 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112670 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-kubelet\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112718 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-ovn\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.113827 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112820 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-ovn\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-var-lib-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112903 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktm4\" (UniqueName: \"kubernetes.io/projected/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-kube-api-access-5ktm4\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-var-lib-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.112983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.113028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-tmp\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.113074 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.113171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-system-cni-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.113333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-env-overrides\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.113387 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-config\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.113485 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.113531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-system-cni-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.114299 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-script-lib\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.114385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.114333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06841536-c400-4c0f-8d46-5d8e24ba9c6d-konnectivity-ca\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:33.114826 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.114396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovn-node-metrics-cert\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.115468 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.115440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-tmp\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.115537 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.115504 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06841536-c400-4c0f-8d46-5d8e24ba9c6d-agent-certs\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:33.116727 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.116707 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-etc-tuned\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.180015 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.179984 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fjt\" (UniqueName: \"kubernetes.io/projected/0f0cd7d8-5868-443d-88a4-8d2e464631e9-kube-api-access-h9fjt\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.180015 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.180003 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmjb\" (UniqueName: \"kubernetes.io/projected/38dd707d-5338-4e99-a3b4-49380731aebb-kube-api-access-szmjb\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.180366 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.180348 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktm4\" (UniqueName: \"kubernetes.io/projected/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-kube-api-access-5ktm4\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.180998 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.180980 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rfm\" (UniqueName: \"kubernetes.io/projected/4322a265-eccb-4e0b-89e3-8844e35f3b6e-kube-api-access-p5rfm\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:33.214111 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.214089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:37:33.230343 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.230306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:37:33.248690 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.248665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:37:33.254776 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.254751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xwzz" Nov 03 18:37:33.265908 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.265886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:37:33.286000 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.285979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:37:33.288817 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.288794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:37:33.298750 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.298732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:37:33.307421 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.307402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:33.336589 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:33.336526 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06841536_c400_4c0f_8d46_5d8e24ba9c6d.slice/crio-f0151277ce74115b945de7f56e6b6d52025bcfc709c27643efe0fd1831fb107c WatchSource:0}: Error finding container f0151277ce74115b945de7f56e6b6d52025bcfc709c27643efe0fd1831fb107c: Status 404 returned error can't find the container with id f0151277ce74115b945de7f56e6b6d52025bcfc709c27643efe0fd1831fb107c Nov 03 18:37:33.337086 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:33.337060 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02282d0e3b70f0c274312efbaf7c57b7.slice/crio-437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec WatchSource:0}: Error finding container 437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec: Status 404 returned error can't find the container with id 437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec Nov 03 18:37:33.338150 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:33.338131 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a04c408b2b26c5648ca41d83d5e6490.slice/crio-7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e WatchSource:0}: Error finding container 7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e: Status 404 returned error can't find the container with id 7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e Nov 03 18:37:33.342214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.342200 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:37:33.616608 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.616528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:33.616608 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.616568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:33.616608 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.616587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616691 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616706 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616720 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616743 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616758 2579 projected.go:196] Error preparing data for projected volume kube-api-access-c5zw8 for pod openshift-network-diagnostics/network-check-target-gl7nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616773 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret podName:4d25cdfb-758e-47fa-adbc-8f6953a7be99 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:34.616754844 +0000 UTC m=+3.419790870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret") pod "global-pull-secret-syncer-2np47" (UID: "4d25cdfb-758e-47fa-adbc-8f6953a7be99") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616794 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs podName:4322a265-eccb-4e0b-89e3-8844e35f3b6e nodeName:}" failed. No retries permitted until 2026-11-03 18:37:34.616784313 +0000 UTC m=+3.419820331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs") pod "network-metrics-daemon-df7m6" (UID: "4322a265-eccb-4e0b-89e3-8844e35f3b6e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:33.617279 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:33.616806 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8 podName:aedaa821-137b-4371-9a23-c4a2890cd484 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:34.61680051 +0000 UTC m=+3.419836527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5zw8" (UniqueName: "kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8") pod "network-check-target-gl7nv" (UID: "aedaa821-137b-4371-9a23-c4a2890cd484") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:33.789021 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.788960 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"02282d0e3b70f0c274312efbaf7c57b7","Type":"ContainerStarted","Data":"437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec"} Nov 03 18:37:33.789195 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.789035 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"3a04c408b2b26c5648ca41d83d5e6490","Type":"ContainerStarted","Data":"7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e"} Nov 03 18:37:33.790153 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.790129 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sdxh" event={"ID":"06841536-c400-4c0f-8d46-5d8e24ba9c6d","Type":"ContainerStarted","Data":"f0151277ce74115b945de7f56e6b6d52025bcfc709c27643efe0fd1831fb107c"} Nov 03 18:37:33.843982 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.843952 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:33.868522 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.868469 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:33.899246 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:33.899227 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:37:34.057385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.057327 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:33 +0000 UTC" deadline="2028-04-08 20:33:37.94201828 +0000 UTC" Nov 03 18:37:34.057385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.057377 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12529h56m3.884646046s" Nov 03 18:37:34.086735 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.086712 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dca58b4_bcb5_4e7e_8eaa_d42b381ef76d.slice/crio-29184f50356ca7962a899b5bca89c9ddd42bda35c351bdae2f28c6c4b4637a81 WatchSource:0}: Error finding container 29184f50356ca7962a899b5bca89c9ddd42bda35c351bdae2f28c6c4b4637a81: Status 404 returned error can't find the container with id 29184f50356ca7962a899b5bca89c9ddd42bda35c351bdae2f28c6c4b4637a81 Nov 03 18:37:34.096753 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.096730 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7933f6c_3101_4ee6_af60_ecec7ac89f03.slice/crio-838b57331b9444f84e1f368cfe14a073314d10adf7cf6628368a2dc980ae15e9 WatchSource:0}: Error finding container 838b57331b9444f84e1f368cfe14a073314d10adf7cf6628368a2dc980ae15e9: Status 404 returned error can't find the container with id 838b57331b9444f84e1f368cfe14a073314d10adf7cf6628368a2dc980ae15e9 Nov 03 18:37:34.153890 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.153865 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dd707d_5338_4e99_a3b4_49380731aebb.slice/crio-8951a80fdfce1b734d8cfa9f9e3caa7bf7144cbbefa1ae50925b9d2ac0cdb85b WatchSource:0}: Error finding container 8951a80fdfce1b734d8cfa9f9e3caa7bf7144cbbefa1ae50925b9d2ac0cdb85b: Status 404 returned error can't find the container with id 8951a80fdfce1b734d8cfa9f9e3caa7bf7144cbbefa1ae50925b9d2ac0cdb85b Nov 03 18:37:34.189940 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.189918 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0cd7d8_5868_443d_88a4_8d2e464631e9.slice/crio-dcf0125738ee90ff43086a2b7df349ef46b378de20c24a42edb70e7bdc5597ab WatchSource:0}: Error finding container dcf0125738ee90ff43086a2b7df349ef46b378de20c24a42edb70e7bdc5597ab: Status 404 returned error can't find the container with id dcf0125738ee90ff43086a2b7df349ef46b378de20c24a42edb70e7bdc5597ab Nov 03 18:37:34.192091 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.192065 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14054e49_309c_46d4_ba51_5a2ad669f1a0.slice/crio-5d0497e7f8739ba2a8ba132568e7fe4b1f26ac035607c7556b4b9b5aa225b60d WatchSource:0}: Error finding container 5d0497e7f8739ba2a8ba132568e7fe4b1f26ac035607c7556b4b9b5aa225b60d: Status 404 returned error can't find the container with id 5d0497e7f8739ba2a8ba132568e7fe4b1f26ac035607c7556b4b9b5aa225b60d Nov 03 18:37:34.194822 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.194802 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736cce59_1d70_4b20_975d_58d36563698f.slice/crio-d384cc8598323a5e878607b219947aa1082bbc15298d13d8e7d5f8b22026d53a WatchSource:0}: Error finding container d384cc8598323a5e878607b219947aa1082bbc15298d13d8e7d5f8b22026d53a: Status 404 returned error can't find the container with id d384cc8598323a5e878607b219947aa1082bbc15298d13d8e7d5f8b22026d53a Nov 03 18:37:34.217704 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.217680 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c91d4f5_0557_4eda_abb2_9f49e237d4b5.slice/crio-9baa51842883a506edc8c8551b6d5b44f1f20b418357b3c0dcb08494ea2e864e WatchSource:0}: Error finding container 9baa51842883a506edc8c8551b6d5b44f1f20b418357b3c0dcb08494ea2e864e: Status 404 returned error can't find the container with id 9baa51842883a506edc8c8551b6d5b44f1f20b418357b3c0dcb08494ea2e864e Nov 03 18:37:34.218268 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:37:34.218234 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f7e70a_abc0_49c1_a7a5_15ef4ff91925.slice/crio-8f12979913fac9cba1164846d6b162a29116dcb595d0e8c61a08fb9d95f5de31 WatchSource:0}: Error finding container 8f12979913fac9cba1164846d6b162a29116dcb595d0e8c61a08fb9d95f5de31: Status 404 returned error can't find the container with id 8f12979913fac9cba1164846d6b162a29116dcb595d0e8c61a08fb9d95f5de31 Nov 03 18:37:34.626105 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.626022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:34.626105 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.626077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.626139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626205 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626277 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs podName:4322a265-eccb-4e0b-89e3-8844e35f3b6e nodeName:}" failed. No retries permitted until 2026-11-03 18:37:36.626256213 +0000 UTC m=+5.429292249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs") pod "network-metrics-daemon-df7m6" (UID: "4322a265-eccb-4e0b-89e3-8844e35f3b6e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626287 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626314 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626326 2579 projected.go:196] Error preparing data for projected volume kube-api-access-c5zw8 for pod openshift-network-diagnostics/network-check-target-gl7nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626352 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626380 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8 podName:aedaa821-137b-4371-9a23-c4a2890cd484 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:36.626361952 +0000 UTC m=+5.429397972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5zw8" (UniqueName: "kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8") pod "network-check-target-gl7nv" (UID: "aedaa821-137b-4371-9a23-c4a2890cd484") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:34.626733 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.626415 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret podName:4d25cdfb-758e-47fa-adbc-8f6953a7be99 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:36.626404077 +0000 UTC m=+5.429440096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret") pod "global-pull-secret-syncer-2np47" (UID: "4d25cdfb-758e-47fa-adbc-8f6953a7be99") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:34.780790 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.780760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:34.780959 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.780887 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:34.781017 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.780973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:34.781074 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.781055 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:34.781141 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.781128 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:34.781235 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:34.781218 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:34.793441 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.793407 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" event={"ID":"38dd707d-5338-4e99-a3b4-49380731aebb","Type":"ContainerStarted","Data":"8951a80fdfce1b734d8cfa9f9e3caa7bf7144cbbefa1ae50925b9d2ac0cdb85b"} Nov 03 18:37:34.795302 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.795229 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" event={"ID":"a7933f6c-3101-4ee6-af60-ecec7ac89f03","Type":"ContainerStarted","Data":"838b57331b9444f84e1f368cfe14a073314d10adf7cf6628368a2dc980ae15e9"} Nov 03 18:37:34.796968 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.796941 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xwzz" event={"ID":"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d","Type":"ContainerStarted","Data":"29184f50356ca7962a899b5bca89c9ddd42bda35c351bdae2f28c6c4b4637a81"} Nov 03 18:37:34.798715 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.798652 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"8f12979913fac9cba1164846d6b162a29116dcb595d0e8c61a08fb9d95f5de31"} Nov 03 18:37:34.800637 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.800611 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p2wnl" event={"ID":"2c91d4f5-0557-4eda-abb2-9f49e237d4b5","Type":"ContainerStarted","Data":"9baa51842883a506edc8c8551b6d5b44f1f20b418357b3c0dcb08494ea2e864e"} Nov 03 18:37:34.802457 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.802434 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-blnln" event={"ID":"736cce59-1d70-4b20-975d-58d36563698f","Type":"ContainerStarted","Data":"d384cc8598323a5e878607b219947aa1082bbc15298d13d8e7d5f8b22026d53a"} Nov 03 18:37:34.805078 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.805048 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-npkpz" event={"ID":"14054e49-309c-46d4-ba51-5a2ad669f1a0","Type":"ContainerStarted","Data":"5d0497e7f8739ba2a8ba132568e7fe4b1f26ac035607c7556b4b9b5aa225b60d"} Nov 03 18:37:34.806602 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:34.806582 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerStarted","Data":"dcf0125738ee90ff43086a2b7df349ef46b378de20c24a42edb70e7bdc5597ab"} Nov 03 18:37:35.058562 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:35.058481 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:33 +0000 UTC" deadline="2028-07-09 22:26:04.906519783 +0000 UTC" Nov 03 18:37:35.058562 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:35.058524 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14739h48m29.848000492s" Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:36.643609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:36.643668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:36.643701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.643863 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.643924 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret podName:4d25cdfb-758e-47fa-adbc-8f6953a7be99 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.643906472 +0000 UTC m=+9.446942497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret") pod "global-pull-secret-syncer-2np47" (UID: "4d25cdfb-758e-47fa-adbc-8f6953a7be99") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.644011 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.644024 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.644037 2579 projected.go:196] Error preparing data for projected volume kube-api-access-c5zw8 for pod openshift-network-diagnostics/network-check-target-gl7nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.644083 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8 podName:aedaa821-137b-4371-9a23-c4a2890cd484 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.644071951 +0000 UTC m=+9.447107973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5zw8" (UniqueName: "kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8") pod "network-check-target-gl7nv" (UID: "aedaa821-137b-4371-9a23-c4a2890cd484") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.644143 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:36.646789 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.644174 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs podName:4322a265-eccb-4e0b-89e3-8844e35f3b6e nodeName:}" failed. No retries permitted until 2026-11-03 18:37:40.64416336 +0000 UTC m=+9.447199384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs") pod "network-metrics-daemon-df7m6" (UID: "4322a265-eccb-4e0b-89e3-8844e35f3b6e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:36.781374 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:36.781341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:36.781551 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.781475 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:36.781551 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:36.781535 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:36.781670 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.781591 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:36.781670 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:36.781633 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:36.781806 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:36.781708 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:37.834374 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:37.833160 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5sdxh" event={"ID":"06841536-c400-4c0f-8d46-5d8e24ba9c6d","Type":"ContainerStarted","Data":"7cc5c1e155344ec7991b8078641bfd14f20ee34c106c84b3932114af4a306f43"} Nov 03 18:37:37.837941 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:37.837226 2579 generic.go:356] "Generic (PLEG): container finished" podID="02282d0e3b70f0c274312efbaf7c57b7" containerID="7b5255ee7e685a3b3da48de2e61da2c46207dc8f5c1e26584602cde5003f793f" exitCode=0 Nov 03 18:37:37.837941 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:37.837272 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"02282d0e3b70f0c274312efbaf7c57b7","Type":"ContainerDied","Data":"7b5255ee7e685a3b3da48de2e61da2c46207dc8f5c1e26584602cde5003f793f"} Nov 03 18:37:38.093707 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.093647 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5sdxh" podStartSLOduration=3.049644517 podStartE2EDuration="6.093624779s" totalImagesPullingTime="3.043980262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:37.963704961 +0000 UTC m=+6.766741004" watchObservedRunningTime="2026-11-03 18:37:38.093624779 +0000 UTC m=+6.896660824" Nov 03 18:37:38.781407 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.781372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:38.781619 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:38.781527 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:38.781619 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.781612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:38.781776 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:38.781698 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:38.781844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.781811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:38.781939 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:38.781913 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:38.844293 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.843559 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"3a04c408b2b26c5648ca41d83d5e6490","Type":"ContainerStarted","Data":"4a3bcbb778ed34a4aed4ae1a15acf9acdf6c1cfe83798121c3b4459eddfb5c8d"} Nov 03 18:37:38.847424 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.847393 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="a00cc96a751139d44617e4956317a8d1b36b4da6268b7a380f340fc88ae3e08e" exitCode=0 Nov 03 18:37:38.847800 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.847707 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerDied","Data":"a00cc96a751139d44617e4956317a8d1b36b4da6268b7a380f340fc88ae3e08e"} Nov 03 18:37:38.852521 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.852461 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"02282d0e3b70f0c274312efbaf7c57b7","Type":"ContainerStarted","Data":"8cb32b69011f638c75aa999d73e826e2f9f24caa066b4f684bcbaf3c7e7dbca6"} Nov 03 18:37:38.894245 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:38.894184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:39.110782 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:39.109224 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" podStartSLOduration=7.109200921 podStartE2EDuration="7.109200921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:38.972412461 +0000 UTC m=+7.775448506" watchObservedRunningTime="2026-11-03 18:37:39.109200921 +0000 UTC m=+7.912236968" Nov 03 18:37:39.219559 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:39.219468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:39.247530 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:39.247471 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" podStartSLOduration=7.247456752 podStartE2EDuration="7.247456752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:39.246844388 +0000 UTC m=+8.049880433" watchObservedRunningTime="2026-11-03 18:37:39.247456752 +0000 UTC m=+8.050492794" Nov 03 18:37:40.065203 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.065054 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:40.066912 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.066239 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:40.682218 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.682186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:40.682398 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.682234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:40.682398 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.682295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:40.682530 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682435 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:40.682530 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682454 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:40.682530 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682466 2579 projected.go:196] Error preparing data for projected volume kube-api-access-c5zw8 for pod openshift-network-diagnostics/network-check-target-gl7nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:40.682530 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682521 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8 podName:aedaa821-137b-4371-9a23-c4a2890cd484 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:48.6825026 +0000 UTC m=+17.485538621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5zw8" (UniqueName: "kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8") pod "network-check-target-gl7nv" (UID: "aedaa821-137b-4371-9a23-c4a2890cd484") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:40.682784 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682598 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:40.682784 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682635 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs podName:4322a265-eccb-4e0b-89e3-8844e35f3b6e nodeName:}" failed. No retries permitted until 2026-11-03 18:37:48.68262133 +0000 UTC m=+17.485657355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs") pod "network-metrics-daemon-df7m6" (UID: "4322a265-eccb-4e0b-89e3-8844e35f3b6e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:40.682784 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682692 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:40.682784 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.682722 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret podName:4d25cdfb-758e-47fa-adbc-8f6953a7be99 nodeName:}" failed. No retries permitted until 2026-11-03 18:37:48.682711512 +0000 UTC m=+17.485747537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret") pod "global-pull-secret-syncer-2np47" (UID: "4d25cdfb-758e-47fa-adbc-8f6953a7be99") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:40.780820 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.780738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:40.780987 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.780902 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:40.780987 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.780945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:40.781123 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.781023 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:40.781123 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.781080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:40.781270 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:40.781248 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:40.861409 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.861372 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-npkpz" event={"ID":"14054e49-309c-46d4-ba51-5a2ad669f1a0","Type":"ContainerStarted","Data":"a132a954485f454ff075aef1f100b59a708078dfcb988e25beba080a5f253b3c"} Nov 03 18:37:40.866000 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.865966 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" event={"ID":"a7933f6c-3101-4ee6-af60-ecec7ac89f03","Type":"ContainerStarted","Data":"324eb603529e8805042ce3c03a8c69329b14f58a6fe183dd93efd33f969814c1"} Nov 03 18:37:40.866130 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.866072 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:40.866672 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:40.866652 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:37:41.104647 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:41.104555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:41.131194 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:41.131140 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-npkpz" podStartSLOduration=3.476283894 podStartE2EDuration="9.131119602s" totalImagesPullingTime="5.654835708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:41.002325889 +0000 UTC m=+9.805361932" watchObservedRunningTime="2026-11-03 18:37:41.131119602 +0000 UTC m=+9.934155651" Nov 03 18:37:41.393238 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:41.393164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:41.872018 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:41.871981 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xwzz" event={"ID":"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d","Type":"ContainerStarted","Data":"8f36551a46fdd13f2a0188cb4e4a8484595a99e507dde09049bc72ecd0d70e15"} Nov 03 18:37:42.005740 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.005614 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8xwzz" podStartSLOduration=2.509570339 podStartE2EDuration="10.00559372s" totalImagesPullingTime="7.496023381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:42.004653257 +0000 UTC m=+10.807689299" watchObservedRunningTime="2026-11-03 18:37:42.00559372 +0000 UTC m=+10.808629764" Nov 03 18:37:42.212217 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.212132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:42.490638 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.490556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:42.712700 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.712665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:42.760287 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.760264 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:37:42.781039 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.781012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:42.781167 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:42.781137 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:42.781271 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.781225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:42.781327 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:42.781308 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:42.781384 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.781372 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:42.781476 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:42.781455 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:42.792665 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.792560 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:37:42.76028163Z","UUID":"17738b28-45de-4449-b981-728ef652e20e","Handler":null,"Name":"","Endpoint":""} Nov 03 18:37:42.794598 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.794575 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:37:42.794686 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.794612 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:37:42.876220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:42.876172 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" event={"ID":"a7933f6c-3101-4ee6-af60-ecec7ac89f03","Type":"ContainerStarted","Data":"df2f9f2887ba6d0f3c8bd6bddc86e7912230e27fc575a182899d8b30ce160ccd"} Nov 03 18:37:43.880967 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:43.880872 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p2wnl" event={"ID":"2c91d4f5-0557-4eda-abb2-9f49e237d4b5","Type":"ContainerStarted","Data":"e502884455295bb5d9ba0d6d706aa3e79257595b5bc841e2671e55d05cad5468"} Nov 03 18:37:44.781553 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:44.781300 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:44.781721 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:44.781323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:44.781721 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:44.781636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:44.781721 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:44.781328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:44.781876 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:44.781712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:44.781876 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:44.781755 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:44.885246 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:44.885210 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-blnln" event={"ID":"736cce59-1d70-4b20-975d-58d36563698f","Type":"ContainerStarted","Data":"04e95927a1b8298a9f829fad44d577c230d4057141da9b48f29f8edf1af1e84d"} Nov 03 18:37:44.887160 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:44.887134 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="6de743d19d8ffe42aa3df62270b6b06b4b61adf31781e73c29c0249d58190412" exitCode=0 Nov 03 18:37:44.887290 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:44.887214 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerDied","Data":"6de743d19d8ffe42aa3df62270b6b06b4b61adf31781e73c29c0249d58190412"} Nov 03 18:37:44.890090 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:44.889742 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" event={"ID":"a7933f6c-3101-4ee6-af60-ecec7ac89f03","Type":"ContainerStarted","Data":"48a2953485e6c284fdf44dc80b53a37a13f9c5dfc8fef30f930c7027c31a2233"} Nov 03 18:37:45.006092 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:45.006045 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-p2wnl" podStartSLOduration=4.465139077 podStartE2EDuration="13.006031564s" totalImagesPullingTime="8.540892487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:44.001582861 +0000 UTC m=+12.804618903" watchObservedRunningTime="2026-11-03 18:37:45.006031564 +0000 UTC m=+13.809067644" Nov 03 18:37:45.128946 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:45.128848 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" podStartSLOduration=3.1559470689999998 podStartE2EDuration="13.128799549s" totalImagesPullingTime="9.97285248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:45.128411441 +0000 UTC m=+13.931447484" watchObservedRunningTime="2026-11-03 18:37:45.128799549 +0000 UTC m=+13.931835589" Nov 03 18:37:45.129221 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:45.129181 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-blnln" podStartSLOduration=4.684673711 podStartE2EDuration="13.129169693s" totalImagesPullingTime="8.444495982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:45.006322882 +0000 UTC m=+13.809358921" watchObservedRunningTime="2026-11-03 18:37:45.129169693 +0000 UTC m=+13.932205734" Nov 03 18:37:45.199145 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:45.199117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:45.410936 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:45.410849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:45.895103 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:45.895030 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" event={"ID":"38dd707d-5338-4e99-a3b4-49380731aebb","Type":"ContainerStarted","Data":"8216acc5f80940c7d0fcc8eea58a20b80efa78b234851f06ce059dc99836bf49"} Nov 03 18:37:46.780894 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:46.780854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:46.780894 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:46.780885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:46.781106 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:46.780855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:46.781106 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:46.780982 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:46.781106 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:46.781080 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:46.781234 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:46.781149 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:46.873891 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:46.873860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:46.898190 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:46.898160 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="ba655e881d0b48c7fcfdcff3b6927b83ad93329fc0dd3866a6695d0b19c2b4b4" exitCode=0 Nov 03 18:37:46.898867 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:46.898265 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerDied","Data":"ba655e881d0b48c7fcfdcff3b6927b83ad93329fc0dd3866a6695d0b19c2b4b4"} Nov 03 18:37:47.088671 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:47.088590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:47.118096 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:47.118044 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" podStartSLOduration=4.342945257 podStartE2EDuration="15.118028757s" totalImagesPullingTime="10.7750835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:46.017494052 +0000 UTC m=+14.820530096" watchObservedRunningTime="2026-11-03 18:37:47.118028757 +0000 UTC m=+15.921064800" Nov 03 18:37:47.296871 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:47.296830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:48.203413 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.203383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:48.413883 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.413675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:48.747751 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.747712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:48.747911 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.747765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:48.747911 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.747792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:48.747911 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.747895 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:48.747911 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.747903 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:37:48.748122 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.747924 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:37:48.748122 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.747937 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret podName:4d25cdfb-758e-47fa-adbc-8f6953a7be99 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:04.747925633 +0000 UTC m=+33.550961650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret") pod "global-pull-secret-syncer-2np47" (UID: "4d25cdfb-758e-47fa-adbc-8f6953a7be99") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:37:48.748122 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.747936 2579 projected.go:196] Error preparing data for projected volume kube-api-access-c5zw8 for pod openshift-network-diagnostics/network-check-target-gl7nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:48.748122 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.747990 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8 podName:aedaa821-137b-4371-9a23-c4a2890cd484 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:04.747974055 +0000 UTC m=+33.551010073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5zw8" (UniqueName: "kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8") pod "network-check-target-gl7nv" (UID: "aedaa821-137b-4371-9a23-c4a2890cd484") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:37:48.748122 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.748095 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:48.748328 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.748134 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs podName:4322a265-eccb-4e0b-89e3-8844e35f3b6e nodeName:}" failed. No retries permitted until 2026-11-03 18:38:04.748119169 +0000 UTC m=+33.551155187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs") pod "network-metrics-daemon-df7m6" (UID: "4322a265-eccb-4e0b-89e3-8844e35f3b6e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:37:48.780845 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.780815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:48.780845 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.780844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:48.781001 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.780901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:48.781001 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.780908 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:48.781100 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.780995 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:48.781100 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:48.781063 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:48.881526 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.881448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:48.904624 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.904589 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="a71cab2f636fda979a8034ae7e6cfc1486ccfa61bd94c8f9d7db1d561baae5bb" exitCode=0 Nov 03 18:37:48.904795 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:48.904641 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerDied","Data":"a71cab2f636fda979a8034ae7e6cfc1486ccfa61bd94c8f9d7db1d561baae5bb"} Nov 03 18:37:49.110340 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.110262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:49.333803 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.333716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:49.909180 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.908962 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"941c8636c3bd358b7da5398201395306e117927f713940842c29ff978c65f9eb"} Nov 03 18:37:49.909311 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.909180 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"7d7d25c51d7d68521c03d968b6e6a8fd9dd2736a7ed1b93ab3d40eb7825b1bd6"} Nov 03 18:37:49.909311 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.909212 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"2da5cb1296480d01bc4e08c8e64dbaac450953bc6f8eddf870eab1ea9f2c1bbe"} Nov 03 18:37:49.909311 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.909240 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"262c5e42f4f58aca5763dde2486d27aab58688a311fcf70e1e52b6e6e2166ffe"} Nov 03 18:37:49.909311 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.909269 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"9533a004e07700ec702845f542c2fd99799d23f09b40746fbe2f7789d80e6b61"} Nov 03 18:37:49.909311 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:49.909297 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"1786b5ead9ad29c2eb611d7f3bf698331a3a3cb595f6a7d9c104019cd1fcd0a8"} Nov 03 18:37:50.780648 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:50.780616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:50.781209 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:50.780616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:50.781209 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:50.780755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:50.781209 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:50.780767 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:50.781209 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:50.780866 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:50.781209 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:50.780962 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:51.916265 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:51.916228 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="7a742cce491eec5b1b1cad8a77149176846b2e7d645ab77d683612e11d20b874" exitCode=0 Nov 03 18:37:51.916833 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:51.916312 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerDied","Data":"7a742cce491eec5b1b1cad8a77149176846b2e7d645ab77d683612e11d20b874"} Nov 03 18:37:51.919756 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:51.919626 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"26376bd08f45983aeed3bd6604fb4cfd5846984a6cb83250e976e8b79dffb1bc"} Nov 03 18:37:52.017958 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:52.017930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:52.781148 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:52.781107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:52.781148 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:52.781134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:52.781366 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:52.781141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:52.781366 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:52.781232 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:52.781366 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:52.781291 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:52.781366 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:52.781344 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:52.924528 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:52.924500 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="ef69bf03eebe06081e33bcb0ed8f20c0ab0ec7c169ef5f31f12fbf60c1a3318c" exitCode=0 Nov 03 18:37:52.924885 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:52.924556 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerDied","Data":"ef69bf03eebe06081e33bcb0ed8f20c0ab0ec7c169ef5f31f12fbf60c1a3318c"} Nov 03 18:37:53.932118 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:53.932078 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" event={"ID":"46f7e70a-abc0-49c1-a7a5-15ef4ff91925","Type":"ContainerStarted","Data":"f35627eeada63b36021431f39493a7dccc8a9e5ad97575d72a1f5128847c25f9"} Nov 03 18:37:53.936068 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:53.936044 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" event={"ID":"0f0cd7d8-5868-443d-88a4-8d2e464631e9","Type":"ContainerStarted","Data":"43601a19008a6fd5aef5950f6a2a553fe9cc18e0909be9fbf86cc75d26b95873"} Nov 03 18:37:54.165807 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:54.165711 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wlzkp" podStartSLOduration=8.567306747 podStartE2EDuration="22.16567794s" totalImagesPullingTime="13.598371193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:54.165211518 +0000 UTC m=+22.968247564" watchObservedRunningTime="2026-11-03 18:37:54.16567794 +0000 UTC m=+22.968713980" Nov 03 18:37:54.780881 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:54.780852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:54.781030 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:54.780908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:54.781030 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:54.780933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:54.781030 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:54.781019 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:54.781131 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:54.781056 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:54.781131 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:54.781103 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:55.188989 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:55.188742 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" podStartSLOduration=8.080101187 podStartE2EDuration="23.188729275s" totalImagesPullingTime="15.108628088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:37:55.188548385 +0000 UTC m=+23.991584421" watchObservedRunningTime="2026-11-03 18:37:55.188729275 +0000 UTC m=+23.991765314" Nov 03 18:37:55.621296 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:55.621219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:56.781262 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:56.781225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:56.781635 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:56.781225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:56.781635 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:56.781331 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:56.781635 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:56.781233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:56.781635 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:56.781406 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:56.781635 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:56.781508 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:56.905699 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:56.905663 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p2wnl_2c91d4f5-0557-4eda-abb2-9f49e237d4b5/dns-node-resolver/0.log" Nov 03 18:37:57.492904 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:57.492868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:57.898443 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:57.898361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:58.098183 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:58.098157 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-npkpz_14054e49-309c-46d4-ba51-5a2ad669f1a0/node-ca/0.log" Nov 03 18:37:58.102956 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:58.102932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:58.781639 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:58.781608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:37:58.781816 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:58.781617 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:37:58.781816 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:58.781743 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:37:58.781816 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:58.781788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:37:58.781916 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:58.781873 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:37:58.781948 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:37:58.781921 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:37:58.978002 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:58.977970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:59.194066 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:59.193977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:59.400427 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:59.400390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:59.613306 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:59.613223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:59.824609 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:37:59.824559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:00.294443 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.294414 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:00.294945 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.294650 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:38:00.295192 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.295168 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:00.295330 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.295246 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:00.311094 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.311069 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:00.311282 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.311176 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:00.315055 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.315025 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" podUID="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 18:38:00.323509 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.323480 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" podUID="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 18:38:00.750043 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.750009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:00.781099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.781067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:00.781099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.781089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:00.781296 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.781174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:00.781296 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:00.781195 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:00.781296 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:00.781241 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:00.781438 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:00.781322 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:00.960035 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.959999 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" podUID="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 18:38:00.960333 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:00.960310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:01.176063 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:01.175975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:01.380328 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:01.380292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:01.595936 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:01.595841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:01.801098 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:01.801066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:01.962527 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:01.962442 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" podUID="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 18:38:02.781560 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:02.781528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:02.781560 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:02.781561 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:02.782107 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:02.781568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:02.782107 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:02.781658 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:02.782107 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:02.781759 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:02.782107 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:02.781825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:02.980605 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:02.980568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:03.192708 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:03.192621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:03.401461 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:03.401426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:03.614400 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:03.614318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:03.823061 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:03.823031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:04.537823 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:04.537788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:04.757818 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:04.757779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:04.757818 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:04.757822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:04.757875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.757928 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.757983 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.757990 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs podName:4322a265-eccb-4e0b-89e3-8844e35f3b6e nodeName:}" failed. No retries permitted until 2026-11-03 18:38:36.757975937 +0000 UTC m=+65.561011954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs") pod "network-metrics-daemon-df7m6" (UID: "4322a265-eccb-4e0b-89e3-8844e35f3b6e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.757992 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.758002 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.758016 2579 projected.go:196] Error preparing data for projected volume kube-api-access-c5zw8 for pod openshift-network-diagnostics/network-check-target-gl7nv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.758042 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret podName:4d25cdfb-758e-47fa-adbc-8f6953a7be99 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:36.758029183 +0000 UTC m=+65.561065204 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret") pod "global-pull-secret-syncer-2np47" (UID: "4d25cdfb-758e-47fa-adbc-8f6953a7be99") : object "kube-system"/"original-pull-secret" not registered Nov 03 18:38:04.758076 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.758056 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8 podName:aedaa821-137b-4371-9a23-c4a2890cd484 nodeName:}" failed. No retries permitted until 2026-11-03 18:38:36.758049511 +0000 UTC m=+65.561085528 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c5zw8" (UniqueName: "kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8") pod "network-check-target-gl7nv" (UID: "aedaa821-137b-4371-9a23-c4a2890cd484") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 18:38:04.780941 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:04.780912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:04.781043 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:04.780912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:04.781043 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.781022 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:04.781112 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.781095 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:04.781112 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:04.780912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:04.781178 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:04.781164 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:06.020923 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:06.020883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:06.238397 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:06.238362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:06.781374 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:06.781341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:06.781374 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:06.781357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:06.781582 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:06.781385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:06.781582 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:06.781458 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:06.781582 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:06.781491 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:06.781582 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:06.781564 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:06.950166 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:06.950134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:07.163035 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:07.162955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:07.373070 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:07.373037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:07.583935 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:07.583851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:07.794322 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:07.794284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:08.147172 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:08.147136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:08.780854 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:08.780821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:08.781251 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:08.780859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:08.781251 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:08.780867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:08.781251 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:08.780935 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:08.781251 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:08.780997 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:08.781251 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:08.781063 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:09.578426 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:09.578390 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:09.829819 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:09.829743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:10.065350 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:10.065316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:10.272518 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:10.272435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:10.780909 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:10.780884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:10.781099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:10.780909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:10.781099 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:10.780921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:10.781099 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:10.780991 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:10.781099 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:10.781056 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:10.781251 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:10.781101 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:11.099403 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.099319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:11.312113 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.312082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:11.586019 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.585988 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-df7m6"] Nov 03 18:38:11.586187 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.586095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:11.586268 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:11.586183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:11.589402 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.589380 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2np47"] Nov 03 18:38:11.589544 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.589462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:11.589544 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:11.589534 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:11.589889 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.589869 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gl7nv"] Nov 03 18:38:11.589952 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:11.589937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:11.590016 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:11.589997 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:12.240720 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:12.240666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:12.453970 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:12.453876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:12.664522 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:12.664489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:12.781050 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:12.780966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:12.781050 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:12.781000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:12.781261 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:12.781068 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:12.781261 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:12.781152 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:12.878504 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:12.878468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:13.297736 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:13.297547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:13.564405 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:13.564317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:13.777908 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:13.777872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:13.780671 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:13.780655 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:13.780813 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:13.780797 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:14.040430 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:14.040394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:14.248434 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:14.248402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:14.781230 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:14.781193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:14.781230 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:14.781223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:14.781774 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:14.781317 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2np47" podUID="4d25cdfb-758e-47fa-adbc-8f6953a7be99" Nov 03 18:38:14.781774 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:14.781418 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gl7nv" podUID="aedaa821-137b-4371-9a23-c4a2890cd484" Nov 03 18:38:15.754602 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:15.754566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:15.781501 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:15.781481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:15.781828 ip-10-0-139-161 kubenswrapper[2579]: E1103 18:38:15.781601 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-df7m6" podUID="4322a265-eccb-4e0b-89e3-8844e35f3b6e" Nov 03 18:38:15.964447 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:15.964372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:16.430894 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.430869 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeReady" Nov 03 18:38:16.431036 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.430980 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 18:38:16.592859 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.592828 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4fd6d"] Nov 03 18:38:16.594794 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.594779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.595794 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.595766 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-js86h"] Nov 03 18:38:16.597897 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.597877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.603200 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.603177 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p9mbd"] Nov 03 18:38:16.607617 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.606298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.607617 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.607509 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4fd6d"] Nov 03 18:38:16.608549 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.608527 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-js86h"] Nov 03 18:38:16.633915 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.633892 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p9mbd"] Nov 03 18:38:16.650526 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.650500 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:38:16.650632 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.650607 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:38:16.650632 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.650613 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:38:16.650749 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.650650 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zjfk4\"" Nov 03 18:38:16.652108 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.652092 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc7sj\"" Nov 03 18:38:16.656751 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.656736 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:38:16.656825 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.656804 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:38:16.661323 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.661306 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:38:16.661402 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.661364 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88hbk\"" Nov 03 18:38:16.661462 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.661449 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:38:16.661499 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.661450 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:38:16.701578 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.701535 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:38:16.741337 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08020da4-3f4c-4eef-8e5f-126abd177058-config-volume\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.741444 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-crio-socket\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.741444 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-data-volume\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.741444 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08020da4-3f4c-4eef-8e5f-126abd177058-tmp-dir\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.741581 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxjv\" (UniqueName: \"kubernetes.io/projected/1f140b25-ea89-4670-8e35-047a4bd190ac-kube-api-access-qfxjv\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.741581 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08020da4-3f4c-4eef-8e5f-126abd177058-metrics-tls\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.741581 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.741748 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvqf\" (UniqueName: \"kubernetes.io/projected/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-api-access-psvqf\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.741748 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.741851 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f140b25-ea89-4670-8e35-047a4bd190ac-cert\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.741851 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.741778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mqm\" (UniqueName: \"kubernetes.io/projected/08020da4-3f4c-4eef-8e5f-126abd177058-kube-api-access-b4mqm\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.780832 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.780811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:16.780905 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.780813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:16.842852 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.842832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08020da4-3f4c-4eef-8e5f-126abd177058-config-volume\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.842860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-crio-socket\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.842875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-data-volume\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.842892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08020da4-3f4c-4eef-8e5f-126abd177058-tmp-dir\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.842916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxjv\" (UniqueName: \"kubernetes.io/projected/1f140b25-ea89-4670-8e35-047a4bd190ac-kube-api-access-qfxjv\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.842939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08020da4-3f4c-4eef-8e5f-126abd177058-metrics-tls\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.842969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvqf\" (UniqueName: \"kubernetes.io/projected/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-api-access-psvqf\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f140b25-ea89-4670-8e35-047a4bd190ac-cert\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mqm\" (UniqueName: \"kubernetes.io/projected/08020da4-3f4c-4eef-8e5f-126abd177058-kube-api-access-b4mqm\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.843214 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843118 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-crio-socket\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-data-volume\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843407 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08020da4-3f4c-4eef-8e5f-126abd177058-tmp-dir\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843565 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08020da4-3f4c-4eef-8e5f-126abd177058-config-volume\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843728 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843788 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4grw\"" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.843844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843847 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:38:16.844118 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.843846 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkfh8\"" Nov 03 18:38:16.846947 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.846925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08020da4-3f4c-4eef-8e5f-126abd177058-metrics-tls\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.847056 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.847037 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f140b25-ea89-4670-8e35-047a4bd190ac-cert\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.847111 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.847052 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.906228 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.906192 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mqm\" (UniqueName: \"kubernetes.io/projected/08020da4-3f4c-4eef-8e5f-126abd177058-kube-api-access-b4mqm\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.906327 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.906288 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxjv\" (UniqueName: \"kubernetes.io/projected/1f140b25-ea89-4670-8e35-047a4bd190ac-kube-api-access-qfxjv\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.906385 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.906358 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvqf\" (UniqueName: \"kubernetes.io/projected/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-api-access-psvqf\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:16.912217 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.912203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:16.919268 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.919251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-js86h" Nov 03 18:38:16.926407 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:16.926295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:17.348582 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:17.348553 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f140b25_ea89_4670_8e35_047a4bd190ac.slice/crio-8c3438db408273dcae78047b500d35aee3243e107be76b2bc74dd711f50415c0 WatchSource:0}: Error finding container 8c3438db408273dcae78047b500d35aee3243e107be76b2bc74dd711f50415c0: Status 404 returned error can't find the container with id 8c3438db408273dcae78047b500d35aee3243e107be76b2bc74dd711f50415c0 Nov 03 18:38:17.350165 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.350143 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4fd6d"] Nov 03 18:38:17.353142 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:17.353118 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08020da4_3f4c_4eef_8e5f_126abd177058.slice/crio-d271c782a6d0661e6a9a9da46b2e0b775d8de75d0239cfef5e448c90641d7af8 WatchSource:0}: Error finding container d271c782a6d0661e6a9a9da46b2e0b775d8de75d0239cfef5e448c90641d7af8: Status 404 returned error can't find the container with id d271c782a6d0661e6a9a9da46b2e0b775d8de75d0239cfef5e448c90641d7af8 Nov 03 18:38:17.354322 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.354261 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-js86h"] Nov 03 18:38:17.356798 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.356779 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p9mbd"] Nov 03 18:38:17.356885 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:17.356863 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43875fcc_00e2_40f4_b610_f4c8aa6ac7ce.slice/crio-f2aa525220713059f73e53208100102b32390dabc52f8252e668e859f2aa3def WatchSource:0}: Error finding container f2aa525220713059f73e53208100102b32390dabc52f8252e668e859f2aa3def: Status 404 returned error can't find the container with id f2aa525220713059f73e53208100102b32390dabc52f8252e668e859f2aa3def Nov 03 18:38:17.586709 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.586667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:17.780993 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.780888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:17.798931 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.798900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:17.835678 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.835656 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dvl9\"" Nov 03 18:38:17.835678 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.835660 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:38:17.947755 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.947731 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-txxgc"] Nov 03 18:38:17.966413 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.966391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:17.988234 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.988211 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-js86h" event={"ID":"08020da4-3f4c-4eef-8e5f-126abd177058","Type":"ContainerStarted","Data":"d271c782a6d0661e6a9a9da46b2e0b775d8de75d0239cfef5e448c90641d7af8"} Nov 03 18:38:17.998731 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.998701 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9mbd" event={"ID":"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce","Type":"ContainerStarted","Data":"c0f2abca284bd5008aeed65d17d48d67ee37593395c2391319694cec1cc127f7"} Nov 03 18:38:17.998823 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.998747 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9mbd" event={"ID":"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce","Type":"ContainerStarted","Data":"f2aa525220713059f73e53208100102b32390dabc52f8252e668e859f2aa3def"} Nov 03 18:38:17.999847 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:17.999828 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4fd6d" event={"ID":"1f140b25-ea89-4670-8e35-047a4bd190ac","Type":"ContainerStarted","Data":"8c3438db408273dcae78047b500d35aee3243e107be76b2bc74dd711f50415c0"} Nov 03 18:38:18.007352 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.007327 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-txxgc"] Nov 03 18:38:18.009279 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.009254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:18.021559 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.021539 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:38:18.021675 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.021588 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:38:18.021752 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.021713 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 18:38:18.021752 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.021716 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 18:38:18.021835 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.021754 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 18:38:18.021864 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.021847 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vgjjr\"" Nov 03 18:38:18.050094 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.050033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.050178 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.050125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b32f9a9-d324-4501-9c23-4e93b00863aa-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.050232 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.050203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.050287 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.050244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlr8v\" (UniqueName: \"kubernetes.io/projected/9b32f9a9-d324-4501-9c23-4e93b00863aa-kube-api-access-hlr8v\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.150810 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.150785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.150939 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.150822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr8v\" (UniqueName: \"kubernetes.io/projected/9b32f9a9-d324-4501-9c23-4e93b00863aa-kube-api-access-hlr8v\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.150939 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.150872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.150939 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.150933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b32f9a9-d324-4501-9c23-4e93b00863aa-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.151656 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.151634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b32f9a9-d324-4501-9c23-4e93b00863aa-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.153784 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.153766 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.153943 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.153882 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.214192 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.214164 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr8v\" (UniqueName: \"kubernetes.io/projected/9b32f9a9-d324-4501-9c23-4e93b00863aa-kube-api-access-hlr8v\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.222699 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.222663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:18.298932 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.298903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:18.641800 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.641672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:18.698067 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:18.698035 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b32f9a9_d324_4501_9c23_4e93b00863aa.slice/crio-ff3194523636614789fc9d65c78cbd83b2d9b3d623678bfcaab73de797685ad7 WatchSource:0}: Error finding container ff3194523636614789fc9d65c78cbd83b2d9b3d623678bfcaab73de797685ad7: Status 404 returned error can't find the container with id ff3194523636614789fc9d65c78cbd83b2d9b3d623678bfcaab73de797685ad7 Nov 03 18:38:18.698435 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.698415 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-txxgc"] Nov 03 18:38:18.851089 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:18.851054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:19.003728 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:19.003636 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" event={"ID":"9b32f9a9-d324-4501-9c23-4e93b00863aa","Type":"ContainerStarted","Data":"ff3194523636614789fc9d65c78cbd83b2d9b3d623678bfcaab73de797685ad7"} Nov 03 18:38:20.009535 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:20.009456 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9mbd" event={"ID":"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce","Type":"ContainerStarted","Data":"d4e7c5ce6223cab31f8bd247ab38e63895f82f9b4e7322463d4cffe5f8b5436a"} Nov 03 18:38:21.013640 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:21.013611 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-js86h" event={"ID":"08020da4-3f4c-4eef-8e5f-126abd177058","Type":"ContainerStarted","Data":"ca4bc11ef5b8ab6883a46ca3d3ccebebddbb0c9b0647b0d57107e84b1fa3404b"} Nov 03 18:38:21.014023 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:21.013659 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-js86h" event={"ID":"08020da4-3f4c-4eef-8e5f-126abd177058","Type":"ContainerStarted","Data":"89824e5aad91129868eb151077ddcfb5f9a0685644ed1cb90f6406dbea9da865"} Nov 03 18:38:21.014023 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:21.013746 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-js86h" Nov 03 18:38:21.135940 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:21.135888 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-js86h" podStartSLOduration=2.243706817 podStartE2EDuration="5.135873232s" totalImagesPullingTime="2.892166415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:21.135596829 +0000 UTC m=+49.938632868" watchObservedRunningTime="2026-11-03 18:38:21.135873232 +0000 UTC m=+49.938909271" Nov 03 18:38:22.018200 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:22.018165 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" event={"ID":"9b32f9a9-d324-4501-9c23-4e93b00863aa","Type":"ContainerStarted","Data":"90a51091f9994b7c163ae31dae1e24d6bf680b72545462e6ca2b8b8f731819f1"} Nov 03 18:38:22.018664 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:22.018258 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" event={"ID":"9b32f9a9-d324-4501-9c23-4e93b00863aa","Type":"ContainerStarted","Data":"1b87021758ecc3df0ea3c684208f211073c7ad1e4fb5ad93c73f52855e5c4eaf"} Nov 03 18:38:22.019670 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:22.019643 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4fd6d" event={"ID":"1f140b25-ea89-4670-8e35-047a4bd190ac","Type":"ContainerStarted","Data":"bb270d533c52bc1cbf31e9f9fb98bb37443be0555e53514ee9fd2722fcee90d3"} Nov 03 18:38:22.140162 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:22.140096 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" podStartSLOduration=2.565059337 podStartE2EDuration="5.140079219s" totalImagesPullingTime="2.575019882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:22.139422469 +0000 UTC m=+50.942458508" watchObservedRunningTime="2026-11-03 18:38:22.140079219 +0000 UTC m=+50.943115260" Nov 03 18:38:22.259782 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:22.259731 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4fd6d" podStartSLOduration=2.326266679 podStartE2EDuration="6.259716782s" totalImagesPullingTime="3.933450103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:22.259339054 +0000 UTC m=+51.062375096" watchObservedRunningTime="2026-11-03 18:38:22.259716782 +0000 UTC m=+51.062752821" Nov 03 18:38:23.024964 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:23.024922 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9mbd" event={"ID":"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce","Type":"ContainerStarted","Data":"336f1960931bacc23af8277e36fd30e1e20cf5c11f498b577fd829bc5f4e01b0"} Nov 03 18:38:23.147823 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:23.147778 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p9mbd" podStartSLOduration=2.599275457 podStartE2EDuration="7.147763211s" totalImagesPullingTime="4.548487754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:38:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:23.147264844 +0000 UTC m=+51.950300883" watchObservedRunningTime="2026-11-03 18:38:23.147763211 +0000 UTC m=+51.950799250" Nov 03 18:38:24.143949 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:24.143908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:24.709609 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:24.709576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:24.956118 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:24.956080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:25.164215 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.164184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:25.373533 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.373497 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:25.417011 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.416950 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv"] Nov 03 18:38:25.449623 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.449602 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.477542 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.477519 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv"] Nov 03 18:38:25.529269 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.529251 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 18:38:25.529413 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.529296 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b7l8x\"" Nov 03 18:38:25.529413 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.529386 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 18:38:25.557585 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.557562 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vj6dq"] Nov 03 18:38:25.581474 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.581452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.585810 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.585785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:25.607405 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.607385 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.607479 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.607414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf78j\" (UniqueName: \"kubernetes.io/projected/42604250-348d-4471-8d8a-d377e8986eaf-kube-api-access-pf78j\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.607479 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.607433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.607549 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.607489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42604250-348d-4471-8d8a-d377e8986eaf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.630442 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.630420 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r"] Nov 03 18:38:25.646400 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.646379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.682669 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.682609 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 18:38:25.682669 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.682661 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 18:38:25.682828 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.682765 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 18:38:25.682828 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.682820 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nx4fl\"" Nov 03 18:38:25.690795 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.690776 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r"] Nov 03 18:38:25.708276 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf78j\" (UniqueName: \"kubernetes.io/projected/42604250-348d-4471-8d8a-d377e8986eaf-kube-api-access-pf78j\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.708360 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-sys\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708360 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.708360 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-textfile\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708457 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-wtmp\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708490 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42604250-348d-4471-8d8a-d377e8986eaf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.708573 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwdr\" (UniqueName: \"kubernetes.io/projected/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-kube-api-access-wcwdr\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708675 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-metrics-client-ca\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708786 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708837 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-root\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708837 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-accelerators-collector-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.708939 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.708982 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.708952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-tls\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.709178 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.709158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42604250-348d-4471-8d8a-d377e8986eaf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.710864 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.710838 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.710995 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.710979 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.740652 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.740630 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 18:38:25.772811 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.772788 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf78j\" (UniqueName: \"kubernetes.io/projected/42604250-348d-4471-8d8a-d377e8986eaf-kube-api-access-pf78j\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.783349 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.783331 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 18:38:25.783526 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.783511 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rdgcx\"" Nov 03 18:38:25.783592 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.783582 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 18:38:25.798205 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.798186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:25.799230 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.799209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:25.809618 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.809599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwdr\" (UniqueName: \"kubernetes.io/projected/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-kube-api-access-wcwdr\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.809742 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.809636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.809742 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.809713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-accelerators-collector-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.809854 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.809834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.809914 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.809882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.809967 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.809921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-wtmp\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810072 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-metrics-client-ca\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810130 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810077 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-wtmp\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810130 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3c563d-6155-43a3-bff6-5e95b9cee38d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.810130 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-textfile\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810275 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810147 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-accelerators-collector-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810275 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810275 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-root\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810275 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl46\" (UniqueName: \"kubernetes.io/projected/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-api-access-gwl46\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.810275 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-sys\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810275 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-tls\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.810579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810364 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-root\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-sys\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810525 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-metrics-client-ca\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.810796 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.810775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-textfile\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.812513 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.812487 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-tls\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.812653 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.812630 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.890293 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.890271 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwdr\" (UniqueName: \"kubernetes.io/projected/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-kube-api-access-wcwdr\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.911388 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.911359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl46\" (UniqueName: \"kubernetes.io/projected/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-api-access-gwl46\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.911496 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.911399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.911496 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.911425 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.911496 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.911487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.911670 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.911513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.911670 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.911537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3c563d-6155-43a3-bff6-5e95b9cee38d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.911896 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.911874 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3c563d-6155-43a3-bff6-5e95b9cee38d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.912212 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.912192 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.912331 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.912309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.913553 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.913535 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.913679 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.913660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.933116 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.933076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:25.942750 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:25.942727 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07cbb8d_1cfb_425b_8de7_e8ae88befa55.slice/crio-b70863381f7a1036a9f5092af393a28a62331aa84e58616260074abb3311cff7 WatchSource:0}: Error finding container b70863381f7a1036a9f5092af393a28a62331aa84e58616260074abb3311cff7: Status 404 returned error can't find the container with id b70863381f7a1036a9f5092af393a28a62331aa84e58616260074abb3311cff7 Nov 03 18:38:25.975238 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.975212 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl46\" (UniqueName: \"kubernetes.io/projected/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-api-access-gwl46\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:25.995595 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:25.995579 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:26.036103 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:26.036077 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vj6dq" event={"ID":"b07cbb8d-1cfb-425b-8de7-e8ae88befa55","Type":"ContainerStarted","Data":"b70863381f7a1036a9f5092af393a28a62331aa84e58616260074abb3311cff7"} Nov 03 18:38:26.150646 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:26.150617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:26.214390 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:26.214362 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42604250_348d_4471_8d8a_d377e8986eaf.slice/crio-68b82928ff6a8143eb8665e0aefaca9162439400823b8bca55b9d3a40c986a5a WatchSource:0}: Error finding container 68b82928ff6a8143eb8665e0aefaca9162439400823b8bca55b9d3a40c986a5a: Status 404 returned error can't find the container with id 68b82928ff6a8143eb8665e0aefaca9162439400823b8bca55b9d3a40c986a5a Nov 03 18:38:26.216457 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:26.216428 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv"] Nov 03 18:38:26.362309 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:26.362279 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3c563d_6155_43a3_bff6_5e95b9cee38d.slice/crio-bf2f025c6739e1e7caf2ed079bc07ac3280927914e54218da1088b9cd3ddd777 WatchSource:0}: Error finding container bf2f025c6739e1e7caf2ed079bc07ac3280927914e54218da1088b9cd3ddd777: Status 404 returned error can't find the container with id bf2f025c6739e1e7caf2ed079bc07ac3280927914e54218da1088b9cd3ddd777 Nov 03 18:38:26.365093 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:26.365073 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r"] Nov 03 18:38:26.537117 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:26.537061 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:26.828132 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:26.828108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:27.035490 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.035453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:27.039828 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.039800 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" event={"ID":"1f3c563d-6155-43a3-bff6-5e95b9cee38d","Type":"ContainerStarted","Data":"bf2f025c6739e1e7caf2ed079bc07ac3280927914e54218da1088b9cd3ddd777"} Nov 03 18:38:27.041873 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.041852 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" event={"ID":"42604250-348d-4471-8d8a-d377e8986eaf","Type":"ContainerStarted","Data":"6fd6a687fe5d84895119b00d91b396a80e6999b4fdc99f1cbb96dfe51b7177f9"} Nov 03 18:38:27.041986 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.041904 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" event={"ID":"42604250-348d-4471-8d8a-d377e8986eaf","Type":"ContainerStarted","Data":"af1af7cdd70b1d87a7a49a1eb6185e52c0479a6bdfb523291b25c961f82019b6"} Nov 03 18:38:27.041986 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.041934 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" event={"ID":"42604250-348d-4471-8d8a-d377e8986eaf","Type":"ContainerStarted","Data":"68b82928ff6a8143eb8665e0aefaca9162439400823b8bca55b9d3a40c986a5a"} Nov 03 18:38:27.259546 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.259461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:27.490290 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.490243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:27.713027 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.712989 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:27.941828 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.941792 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:38:27.945253 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.945227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:27.958291 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:27.958213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:28.005834 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.005800 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-g98j2"] Nov 03 18:38:28.008202 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.008174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.049384 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.049237 2579 generic.go:356] "Generic (PLEG): container finished" podID="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" containerID="b812a8225a268642ab6dbd772c5c8cc638b136043b91403175371cee0f67aef2" exitCode=0 Nov 03 18:38:28.049384 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.049318 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vj6dq" event={"ID":"b07cbb8d-1cfb-425b-8de7-e8ae88befa55","Type":"ContainerDied","Data":"b812a8225a268642ab6dbd772c5c8cc638b136043b91403175371cee0f67aef2"} Nov 03 18:38:28.109182 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.109155 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 18:38:28.109474 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.109443 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 18:38:28.109630 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.109600 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 18:38:28.109729 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.109714 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 18:38:28.109806 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.109793 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 18:38:28.109934 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.109918 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 18:38:28.110056 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110041 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gfh9q\"" Nov 03 18:38:28.110133 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110120 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 18:38:28.110211 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110197 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 18:38:28.110366 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110350 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jk4jl\"" Nov 03 18:38:28.110487 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110472 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 18:38:28.110597 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110583 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 18:38:28.110692 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110670 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 18:38:28.110768 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.110754 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:38:28.111579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.111562 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g98j2"] Nov 03 18:38:28.127335 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127479 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127479 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127387 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127574 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127574 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxsq\" (UniqueName: \"kubernetes.io/projected/7601adc3-f067-48c5-b556-58b607361367-kube-api-access-2nxsq\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.127574 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxgs4\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127674 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127674 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127764 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127764 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127764 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127865 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127865 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.127865 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7601adc3-f067-48c5-b556-58b607361367-tmp\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.127944 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.127879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.143870 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.143839 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c8cc96c46-k9l2m"] Nov 03 18:38:28.146641 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.146615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.202298 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.202201 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 18:38:28.202441 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.202371 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 18:38:28.202441 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.202420 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 18:38:28.202526 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.202465 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3c2nv2isc8l0t\"" Nov 03 18:38:28.202526 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.202469 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 18:38:28.202526 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.202508 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 18:38:28.202633 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.202556 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r6plj\"" Nov 03 18:38:28.208610 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.208585 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c8cc96c46-k9l2m"] Nov 03 18:38:28.228658 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.228783 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.228783 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.228783 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.228942 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxsq\" (UniqueName: \"kubernetes.io/projected/7601adc3-f067-48c5-b556-58b607361367-kube-api-access-2nxsq\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.228942 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxgs4\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.228942 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.229112 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.228957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.229112 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.229112 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.229112 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.229298 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.229298 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.229298 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7601adc3-f067-48c5-b556-58b607361367-tmp\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.229443 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.230603 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.229636 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.230927 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.230891 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.231051 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.231022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.235440 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.235410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7601adc3-f067-48c5-b556-58b607361367-tmp\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.236309 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.236279 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.237546 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.236425 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.237546 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.236701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.237546 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.236748 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.237546 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.237094 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.240815 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.238350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.242514 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.242489 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.243764 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.243711 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.245106 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.245080 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.295951 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.295909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxgs4\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.296350 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.295919 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxsq\" (UniqueName: \"kubernetes.io/projected/7601adc3-f067-48c5-b556-58b607361367-kube-api-access-2nxsq\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.322207 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.322173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:28.329762 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.329737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-grpc-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.329883 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.329772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.329883 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.329791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.329883 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.329866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.330035 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.329902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.330145 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.330114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c026965c-43dc-41e4-a73a-90ba867305a5-metrics-client-ca\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.330254 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.330154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c026965c-43dc-41e4-a73a-90ba867305a5-tmp\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.330254 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.330175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.330254 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.330192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snlf\" (UniqueName: \"kubernetes.io/projected/c026965c-43dc-41e4-a73a-90ba867305a5-kube-api-access-6snlf\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431310 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-grpc-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c026965c-43dc-41e4-a73a-90ba867305a5-metrics-client-ca\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c026965c-43dc-41e4-a73a-90ba867305a5-tmp\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431539 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.431676 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snlf\" (UniqueName: \"kubernetes.io/projected/c026965c-43dc-41e4-a73a-90ba867305a5-kube-api-access-6snlf\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.432127 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.431916 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c026965c-43dc-41e4-a73a-90ba867305a5-tmp\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.432728 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.432699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c026965c-43dc-41e4-a73a-90ba867305a5-metrics-client-ca\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.434202 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.434179 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.434384 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.434366 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.434520 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.434437 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-grpc-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.434582 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.434530 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.434582 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.434530 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.434872 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.434849 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.494523 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.494502 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snlf\" (UniqueName: \"kubernetes.io/projected/c026965c-43dc-41e4-a73a-90ba867305a5-kube-api-access-6snlf\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.568378 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.568353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:28.776297 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.776214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:28.783569 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:28.783535 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7601adc3_f067_48c5_b556_58b607361367.slice/crio-c268ca90aea9539832c856af0be8630a6c4fec12d6ca9baa4ffcf0bdf0a8f8d9 WatchSource:0}: Error finding container c268ca90aea9539832c856af0be8630a6c4fec12d6ca9baa4ffcf0bdf0a8f8d9: Status 404 returned error can't find the container with id c268ca90aea9539832c856af0be8630a6c4fec12d6ca9baa4ffcf0bdf0a8f8d9 Nov 03 18:38:28.784999 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:28.784965 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-g98j2"] Nov 03 18:38:29.026763 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:29.026670 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4e2269_545e_4c5d_b5f9_3e337a0fa843.slice/crio-6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0 WatchSource:0}: Error finding container 6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0: Status 404 returned error can't find the container with id 6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0 Nov 03 18:38:29.026925 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.026901 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:38:29.054708 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.054670 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" event={"ID":"1f3c563d-6155-43a3-bff6-5e95b9cee38d","Type":"ContainerStarted","Data":"ed8d5e99aba2783416aa32c65fafedbf69188f89d98ffb183ad16a75aa4ff25b"} Nov 03 18:38:29.054819 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.054742 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" event={"ID":"1f3c563d-6155-43a3-bff6-5e95b9cee38d","Type":"ContainerStarted","Data":"62c4814fec045a6d7d04919c3a684987ac840cae17bee01586737fca19b3f657"} Nov 03 18:38:29.054819 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.054777 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" event={"ID":"1f3c563d-6155-43a3-bff6-5e95b9cee38d","Type":"ContainerStarted","Data":"37805ef80647f19575ff9e22efa88ae3ae5d4eb5a4d50971cf3f3b9f8a43e89e"} Nov 03 18:38:29.055939 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.055918 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerStarted","Data":"6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0"} Nov 03 18:38:29.057967 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.057944 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vj6dq" event={"ID":"b07cbb8d-1cfb-425b-8de7-e8ae88befa55","Type":"ContainerStarted","Data":"bce4a5ac1e808ff2201334af05d3addf6f410364b1886c9a89363c8dd7ac3e62"} Nov 03 18:38:29.058067 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.057986 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vj6dq" event={"ID":"b07cbb8d-1cfb-425b-8de7-e8ae88befa55","Type":"ContainerStarted","Data":"1d7c97dd9d5b5ad652425c406e2817a7529db1d8df420e8b1ea597c803b2508d"} Nov 03 18:38:29.060044 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.059957 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" event={"ID":"42604250-348d-4471-8d8a-d377e8986eaf","Type":"ContainerStarted","Data":"ed5aa7f6711a8d31e7fd53f5f887e67c5724f5cd0e1ea8bae44d03087dd42ffa"} Nov 03 18:38:29.061644 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.061546 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g98j2" event={"ID":"7601adc3-f067-48c5-b556-58b607361367","Type":"ContainerStarted","Data":"c268ca90aea9539832c856af0be8630a6c4fec12d6ca9baa4ffcf0bdf0a8f8d9"} Nov 03 18:38:29.148750 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:29.148716 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc026965c_43dc_41e4_a73a_90ba867305a5.slice/crio-a6841c60d946d0f38ad89adbdb2692618243bda2a07114704a48c91394cf3c32 WatchSource:0}: Error finding container a6841c60d946d0f38ad89adbdb2692618243bda2a07114704a48c91394cf3c32: Status 404 returned error can't find the container with id a6841c60d946d0f38ad89adbdb2692618243bda2a07114704a48c91394cf3c32 Nov 03 18:38:29.153052 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.153026 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c8cc96c46-k9l2m"] Nov 03 18:38:29.179371 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.179323 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" podStartSLOduration=2.097650984 podStartE2EDuration="4.179309946s" totalImagesPullingTime="2.081658962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:29.178598676 +0000 UTC m=+57.981634715" watchObservedRunningTime="2026-11-03 18:38:29.179309946 +0000 UTC m=+57.982345985" Nov 03 18:38:29.303959 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.303856 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vj6dq" podStartSLOduration=2.666257156 podStartE2EDuration="4.303844075s" totalImagesPullingTime="1.567100749s" totalInitContainerRuntime="70.48617ms" isStatefulPod=true podCreationTimestamp="2026-11-03 18:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:29.302942916 +0000 UTC m=+58.105978956" watchObservedRunningTime="2026-11-03 18:38:29.303844075 +0000 UTC m=+58.106880113" Nov 03 18:38:29.429616 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.429544 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" podStartSLOduration=2.5936613790000003 podStartE2EDuration="4.429529776s" totalImagesPullingTime="1.835868397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:29.428904187 +0000 UTC m=+58.231940227" watchObservedRunningTime="2026-11-03 18:38:29.429529776 +0000 UTC m=+58.232565815" Nov 03 18:38:29.787716 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:29.787658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:30.019625 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.019573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:30.070762 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.070713 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" event={"ID":"c026965c-43dc-41e4-a73a-90ba867305a5","Type":"ContainerStarted","Data":"a6841c60d946d0f38ad89adbdb2692618243bda2a07114704a48c91394cf3c32"} Nov 03 18:38:30.232384 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.232298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:30.444338 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.444249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:30.654289 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.654244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:30.684875 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.683533 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7db59cdcd6-pqldv"] Nov 03 18:38:30.687284 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.687255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.749844 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.749817 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7db59cdcd6-pqldv"] Nov 03 18:38:30.750087 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.750069 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-flzhh\"" Nov 03 18:38:30.750345 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.750193 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 18:38:30.750345 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.750318 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-jq1vc25cvje4\"" Nov 03 18:38:30.750451 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.750440 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 18:38:30.750488 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.750482 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 18:38:30.750588 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.750550 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 18:38:30.851832 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.851800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-client-certs\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.851970 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.851845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-client-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.851970 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.851932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74f5b2ca-aa33-444a-bd6f-56df2278d099-audit-log\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.852138 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.852116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxcg\" (UniqueName: \"kubernetes.io/projected/74f5b2ca-aa33-444a-bd6f-56df2278d099-kube-api-access-jgxcg\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.852210 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.852165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.852266 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.852236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-metrics-server-audit-profiles\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.852329 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.852307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-tls\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.865838 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.865806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:30.953131 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxcg\" (UniqueName: \"kubernetes.io/projected/74f5b2ca-aa33-444a-bd6f-56df2278d099-kube-api-access-jgxcg\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.953261 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.953261 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-metrics-server-audit-profiles\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.953261 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-tls\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.953261 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-client-certs\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.953261 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-client-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.953590 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74f5b2ca-aa33-444a-bd6f-56df2278d099-audit-log\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.953999 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.953975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.954112 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.954053 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74f5b2ca-aa33-444a-bd6f-56df2278d099-audit-log\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.954384 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.954364 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-metrics-server-audit-profiles\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.956641 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.956613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-tls\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.957381 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.957360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-client-certs\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:30.958278 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:30.958255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-client-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:31.017279 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.017215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxcg\" (UniqueName: \"kubernetes.io/projected/74f5b2ca-aa33-444a-bd6f-56df2278d099-kube-api-access-jgxcg\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:31.019066 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.019028 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:31.022090 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.022068 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-js86h" Nov 03 18:38:31.082558 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.081551 2579 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0" exitCode=0 Nov 03 18:38:31.082558 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.081772 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0"} Nov 03 18:38:31.240823 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.240794 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74f6bf747d-rsr85"] Nov 03 18:38:31.244610 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.244222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.300976 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.300810 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 18:38:31.300976 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.300860 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 18:38:31.300976 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.300895 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 18:38:31.300976 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.300898 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 18:38:31.300976 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.300914 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 18:38:31.301314 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.301114 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2n24p\"" Nov 03 18:38:31.341559 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.341525 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 18:38:31.343360 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.343333 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74f6bf747d-rsr85"] Nov 03 18:38:31.357301 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-serving-certs-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.357421 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.357421 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-metrics-client-ca\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.357519 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.357519 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzv4\" (UniqueName: \"kubernetes.io/projected/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-kube-api-access-fvzv4\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.357618 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.357618 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-federate-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.357860 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.357620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.398802 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:31.398772 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f5b2ca_aa33_444a_bd6f_56df2278d099.slice/crio-3675f99fba9b96d9fd96cde6b7f691313c771cc53a32ee9a42263af6fa36d01f WatchSource:0}: Error finding container 3675f99fba9b96d9fd96cde6b7f691313c771cc53a32ee9a42263af6fa36d01f: Status 404 returned error can't find the container with id 3675f99fba9b96d9fd96cde6b7f691313c771cc53a32ee9a42263af6fa36d01f Nov 03 18:38:31.401999 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.401754 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7db59cdcd6-pqldv"] Nov 03 18:38:31.458017 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.457993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-federate-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-serving-certs-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-metrics-client-ca\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzv4\" (UniqueName: \"kubernetes.io/projected/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-kube-api-access-fvzv4\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.458870 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.458845 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-serving-certs-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.459310 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.459264 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.459409 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.459319 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-metrics-client-ca\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.461516 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.461495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.461944 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.461925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.462140 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.462119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-federate-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.462281 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.462263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.532134 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.532111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzv4\" (UniqueName: \"kubernetes.io/projected/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-kube-api-access-fvzv4\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.564280 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.564226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:31.966560 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.966514 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:31.996461 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:31.996435 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74f6bf747d-rsr85"] Nov 03 18:38:32.087588 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:32.087556 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" event={"ID":"3d64f9eb-a543-4518-8d1d-0a27cac5a88a","Type":"ContainerStarted","Data":"e7ad0f3c2a8f14307e72994f8bde9666335c4e2e6a8ae02fc760677b188605b0"} Nov 03 18:38:32.089036 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:32.089009 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" event={"ID":"74f5b2ca-aa33-444a-bd6f-56df2278d099","Type":"ContainerStarted","Data":"3675f99fba9b96d9fd96cde6b7f691313c771cc53a32ee9a42263af6fa36d01f"} Nov 03 18:38:32.304653 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:32.304610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:32.570384 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:32.570323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:32.843014 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:32.842938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:33.050002 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:33.049927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:33.096810 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:33.096732 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" event={"ID":"c026965c-43dc-41e4-a73a-90ba867305a5","Type":"ContainerStarted","Data":"caf501eb44ac4080a6545e7082ab9a8a67d5bdb573bf0fdec18c10656271f0f5"} Nov 03 18:38:33.096944 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:33.096830 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" event={"ID":"c026965c-43dc-41e4-a73a-90ba867305a5","Type":"ContainerStarted","Data":"34f7cee0c403ef87f16835c9db2cc70b4f2429c7f1fa9b74ad8d48814593899f"} Nov 03 18:38:33.096944 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:33.096870 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" event={"ID":"c026965c-43dc-41e4-a73a-90ba867305a5","Type":"ContainerStarted","Data":"9e8b9681c91b48817d4cad8f76ff91f8a6ba5bec266081ee7cda06e4003decf7"} Nov 03 18:38:34.106579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.106486 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" event={"ID":"c026965c-43dc-41e4-a73a-90ba867305a5","Type":"ContainerStarted","Data":"33afcfb1785b08664cb7bf3f99e541bcb23a5891ab74098d6223e0dea065860c"} Nov 03 18:38:34.106579 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.106560 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" event={"ID":"c026965c-43dc-41e4-a73a-90ba867305a5","Type":"ContainerStarted","Data":"99c08921e84fcdc933b04c1416f8b6e50ad073481b5d68ef276b6a182fff5d0c"} Nov 03 18:38:34.107134 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.106594 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" event={"ID":"c026965c-43dc-41e4-a73a-90ba867305a5","Type":"ContainerStarted","Data":"ae7af366d9e094c7afe41d7b6a17060baa8590fd94bcdeaa36b7a6a7f8825f7c"} Nov 03 18:38:34.107134 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.106637 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:34.828247 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.828172 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" podStartSLOduration=2.354092135 podStartE2EDuration="6.828152043s" totalImagesPullingTime="4.474059908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:34.24091728 +0000 UTC m=+63.043953320" watchObservedRunningTime="2026-11-03 18:38:34.828152043 +0000 UTC m=+63.631188083" Nov 03 18:38:34.829145 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.829119 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:38:34.840670 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.832574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.847255 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.847040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.895988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.896841 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.897725 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.897725 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.897725 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.897725 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.896420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smrrx\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.994373 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.994340 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 18:38:34.994977 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.994895 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 18:38:34.994977 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.994921 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 18:38:34.994977 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.994943 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 18:38:34.994977 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.994904 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 18:38:34.995837 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.995813 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 18:38:34.996185 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.996167 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 18:38:34.996244 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.996182 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9c3kkmaf6dql\"" Nov 03 18:38:34.996479 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.996462 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 18:38:34.996950 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.996931 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 18:38:34.997199 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.997181 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 18:38:34.997745 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.997393 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 18:38:34.997745 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.997631 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 18:38:34.997913 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.997810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.997913 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.997856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998014 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.997913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998097 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998198 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998252 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998299 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998441 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998495 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998435 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 18:38:34.998495 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998605 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998655 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smrrx\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:34.998716 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.998872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.999018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.999048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.999077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.999247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.999286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.999442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:34.999897 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.001367 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d6l69\"" Nov 03 18:38:35.004737 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.002360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.005267 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.005192 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:38:35.007165 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.006616 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.016986 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.016960 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.017077 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.017056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.017145 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.017079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.017278 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.017253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.017608 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.017558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.017608 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.017571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.017989 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.017967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.019648 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.019035 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.019648 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.019193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.019648 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.019510 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.019648 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.019601 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.020092 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.019997 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.020480 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.020455 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.021254 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.021233 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.021709 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.021666 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.022229 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.022187 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.023483 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.023461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.076666 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.074910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smrrx\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.117703 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.117652 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerStarted","Data":"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840"} Nov 03 18:38:35.118220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.117762 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerStarted","Data":"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261"} Nov 03 18:38:35.118220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.117808 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerStarted","Data":"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999"} Nov 03 18:38:35.118220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.117851 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerStarted","Data":"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7"} Nov 03 18:38:35.118220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.117888 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerStarted","Data":"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446"} Nov 03 18:38:35.118220 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.117925 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerStarted","Data":"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336"} Nov 03 18:38:35.120863 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.119871 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" event={"ID":"3d64f9eb-a543-4518-8d1d-0a27cac5a88a","Type":"ContainerStarted","Data":"394d96a6bc07ac559eba1ae7610e32689f65d16a9dca686cb5118c56f498b107"} Nov 03 18:38:35.177808 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.177784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:35.621420 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:35.621395 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74be8f9c_362e_4589_b78f_d72f46a83ebc.slice/crio-68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478 WatchSource:0}: Error finding container 68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478: Status 404 returned error can't find the container with id 68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478 Nov 03 18:38:35.624121 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.624072 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.754715907 podStartE2EDuration="8.624054006s" totalImagesPullingTime="4.848166174s" totalInitContainerRuntime="21.171925ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:35.303191095 +0000 UTC m=+64.106227134" watchObservedRunningTime="2026-11-03 18:38:35.624054006 +0000 UTC m=+64.427090047" Nov 03 18:38:35.625090 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:35.625067 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:38:36.126572 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.126533 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" event={"ID":"3d64f9eb-a543-4518-8d1d-0a27cac5a88a","Type":"ContainerStarted","Data":"e635132c33f85107c5d2eb9fef0e0f32a6c59c2904ddc79b75d21a0a17c4898c"} Nov 03 18:38:36.127077 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.126607 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" event={"ID":"3d64f9eb-a543-4518-8d1d-0a27cac5a88a","Type":"ContainerStarted","Data":"66d6b97313c3bcf35fe06d5b5ff6484bef568fd47132a775059bc62fdbb1179c"} Nov 03 18:38:36.128528 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.128499 2579 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" exitCode=0 Nov 03 18:38:36.128670 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.128645 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8"} Nov 03 18:38:36.128779 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.128752 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerStarted","Data":"68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478"} Nov 03 18:38:36.130650 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.130617 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" event={"ID":"74f5b2ca-aa33-444a-bd6f-56df2278d099","Type":"ContainerStarted","Data":"f38174ded3b104d189e0296e81a0aac79965f274a2bc271012375fca82001579"} Nov 03 18:38:36.259525 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.259466 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" podStartSLOduration=2.22787906 podStartE2EDuration="5.25944717s" totalImagesPullingTime="3.03156811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:36.257779893 +0000 UTC m=+65.060815931" watchObservedRunningTime="2026-11-03 18:38:36.25944717 +0000 UTC m=+65.062483210" Nov 03 18:38:36.381809 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.381722 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" podStartSLOduration=2.652950908 podStartE2EDuration="6.381709737s" totalImagesPullingTime="3.728758829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:36.3805134 +0000 UTC m=+65.183549442" watchObservedRunningTime="2026-11-03 18:38:36.381709737 +0000 UTC m=+65.184745773" Nov 03 18:38:36.819800 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.819760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:36.819977 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.819817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:36.819977 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.819866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:36.873944 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.873914 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:38:36.874095 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.873953 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:38:36.874412 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.874394 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:38:36.882989 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.882967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:36.883109 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.883015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:36.934322 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.934299 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:38:36.944340 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.944310 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:36.948115 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.948083 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkfh8\"" Nov 03 18:38:36.954903 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:36.954676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:37.119190 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.119117 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dvl9\"" Nov 03 18:38:37.125313 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.125282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:37.252537 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.252509 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4grw\"" Nov 03 18:38:37.258560 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.258539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:37.334840 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:37.334804 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d25cdfb_758e_47fa_adbc_8f6953a7be99.slice/crio-f254fc156e78afeb61c659d262483dc80e5d815ca8b1ed3cccb00a90cb518191 WatchSource:0}: Error finding container f254fc156e78afeb61c659d262483dc80e5d815ca8b1ed3cccb00a90cb518191: Status 404 returned error can't find the container with id f254fc156e78afeb61c659d262483dc80e5d815ca8b1ed3cccb00a90cb518191 Nov 03 18:38:37.335074 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.334966 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2np47"] Nov 03 18:38:37.505638 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:37.505584 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4322a265_eccb_4e0b_89e3_8844e35f3b6e.slice/crio-c4e1d14c542291d14732309ff716f11f8f2c281f2e8643cfc51d5369b1d81745 WatchSource:0}: Error finding container c4e1d14c542291d14732309ff716f11f8f2c281f2e8643cfc51d5369b1d81745: Status 404 returned error can't find the container with id c4e1d14c542291d14732309ff716f11f8f2c281f2e8643cfc51d5369b1d81745 Nov 03 18:38:37.506630 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.506601 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-df7m6"] Nov 03 18:38:37.646198 ip-10-0-139-161 kubenswrapper[2579]: W1103 18:38:37.645101 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedaa821_137b_4371_9a23_c4a2890cd484.slice/crio-0d509d8dbb2bff095adbd200728378b133b370108ae2d3c4db787ff0b7aeb85f WatchSource:0}: Error finding container 0d509d8dbb2bff095adbd200728378b133b370108ae2d3c4db787ff0b7aeb85f: Status 404 returned error can't find the container with id 0d509d8dbb2bff095adbd200728378b133b370108ae2d3c4db787ff0b7aeb85f Nov 03 18:38:37.646198 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.646127 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gl7nv"] Nov 03 18:38:37.734155 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.734110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:37.964375 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:37.964285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:38.141283 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:38.141246 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-df7m6" event={"ID":"4322a265-eccb-4e0b-89e3-8844e35f3b6e","Type":"ContainerStarted","Data":"c4e1d14c542291d14732309ff716f11f8f2c281f2e8643cfc51d5369b1d81745"} Nov 03 18:38:38.142746 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:38.142715 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2np47" event={"ID":"4d25cdfb-758e-47fa-adbc-8f6953a7be99","Type":"ContainerStarted","Data":"f254fc156e78afeb61c659d262483dc80e5d815ca8b1ed3cccb00a90cb518191"} Nov 03 18:38:38.144059 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:38.144025 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gl7nv" event={"ID":"aedaa821-137b-4371-9a23-c4a2890cd484","Type":"ContainerStarted","Data":"0d509d8dbb2bff095adbd200728378b133b370108ae2d3c4db787ff0b7aeb85f"} Nov 03 18:38:38.756331 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:38.756275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:38.970194 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:38.970148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:39.294282 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:39.294244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:39.513584 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:39.513547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:40.047900 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:40.047850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:40.128560 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:40.128526 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:40.159344 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:40.159306 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-df7m6" event={"ID":"4322a265-eccb-4e0b-89e3-8844e35f3b6e","Type":"ContainerStarted","Data":"6eb8ccc1404f160454164e2cca7335fcc7b3ada989a0461337060426159872c7"} Nov 03 18:38:40.159490 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:40.159362 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-df7m6" event={"ID":"4322a265-eccb-4e0b-89e3-8844e35f3b6e","Type":"ContainerStarted","Data":"38b5270a912e6e47fabbedff881a924105d40bf0c9ea50366c87be3399816661"} Nov 03 18:38:40.299432 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:40.299326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:40.629624 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:40.629589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:40.851817 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:40.851765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:42.176804 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.176725 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerStarted","Data":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} Nov 03 18:38:42.177801 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.176826 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerStarted","Data":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} Nov 03 18:38:42.177801 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.176855 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerStarted","Data":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} Nov 03 18:38:42.177801 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.176893 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerStarted","Data":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} Nov 03 18:38:42.177801 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.176929 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerStarted","Data":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} Nov 03 18:38:42.177801 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.176967 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerStarted","Data":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} Nov 03 18:38:42.456362 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.456255 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-df7m6" podStartSLOduration=68.303784051 podStartE2EDuration="1m10.456240875s" totalImagesPullingTime="2.152456824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:40.43781336 +0000 UTC m=+69.240849401" watchObservedRunningTime="2026-11-03 18:38:42.456240875 +0000 UTC m=+71.259276939" Nov 03 18:38:42.456976 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:42.456930 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.194899623 podStartE2EDuration="8.456912541s" totalImagesPullingTime="5.242116941s" totalInitContainerRuntime="19.895977ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:42.418714497 +0000 UTC m=+71.221750537" watchObservedRunningTime="2026-11-03 18:38:42.456912541 +0000 UTC m=+71.259948581" Nov 03 18:38:43.184201 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:43.184113 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gl7nv" event={"ID":"aedaa821-137b-4371-9a23-c4a2890cd484","Type":"ContainerStarted","Data":"e4b7e54cf1598142347ba4faba4a5a1be19baba989c320d7dfba6fc35cab6cec"} Nov 03 18:38:43.306853 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:43.306791 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gl7nv" podStartSLOduration=66.05861892 podStartE2EDuration="1m11.306772731s" totalImagesPullingTime="5.248153811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:43.305401253 +0000 UTC m=+72.108437294" watchObservedRunningTime="2026-11-03 18:38:43.306772731 +0000 UTC m=+72.109808772" Nov 03 18:38:43.938503 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:43.938464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:44.165700 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:44.165566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:44.188838 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:44.188735 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:45.050054 ip-10-0-139-161 kubenswrapper[2579]: I1103 18:38:45.049941 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:38:45.050142 ip-10-0-139-161 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 18:38:45.057369 ip-10-0-139-161 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 18:38:45.057566 ip-10-0-139-161 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 18:38:45.057753 ip-10-0-139-161 systemd[1]: kubelet.service: Consumed 4.723s CPU time. Nov 03 18:38:45.067371 ip-10-0-139-161 systemd[1]: Starting Kubernetes Kubelet... Nov 03 18:38:45.157754 ip-10-0-139-161 kubenswrapper[5988]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:38:45.157754 ip-10-0-139-161 kubenswrapper[5988]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 18:38:45.157754 ip-10-0-139-161 kubenswrapper[5988]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:38:45.157754 ip-10-0-139-161 kubenswrapper[5988]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 18:38:45.160212 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160116 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.160212 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160208 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.160212 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160213 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.160212 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160218 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160224 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160230 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160234 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160238 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160244 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160248 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160253 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160257 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160262 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160267 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160271 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160276 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160280 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160284 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160289 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160293 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160297 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160302 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160306 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.160481 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160311 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160315 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160320 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160324 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160328 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160333 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160338 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160343 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160347 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160352 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160357 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160361 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160366 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160371 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160376 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160380 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160386 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160391 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160395 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160400 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.161405 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160406 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160411 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160418 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160426 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160431 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160435 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160440 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160445 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160450 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160454 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160459 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160464 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160469 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160473 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160477 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160482 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160487 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160492 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.162338 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160497 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160502 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160506 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160511 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160516 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160520 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160526 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160530 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160535 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160540 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160545 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160550 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160555 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160560 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160565 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160570 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160575 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160579 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160586 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160591 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.163182 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160595 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160600 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160605 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160610 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160615 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160620 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160624 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160629 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160635 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160641 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160647 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160652 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160658 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160662 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160667 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160672 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160676 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160697 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160702 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.164081 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160707 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160711 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160716 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160723 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160728 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.160745 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162084 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162097 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162103 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162108 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162112 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162117 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162122 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162126 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162133 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162137 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162142 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162147 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162151 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162156 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.164963 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162160 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162166 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162170 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162175 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162182 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162186 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162191 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162196 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162201 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162206 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162210 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162216 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162220 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162225 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162230 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162235 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162239 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162244 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162249 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.165888 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162254 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162258 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162263 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162268 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162272 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162277 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162282 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162286 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162291 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162296 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162301 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162305 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162310 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162315 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162319 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162324 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162329 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162333 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162338 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162342 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.166750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162347 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162352 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162358 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162362 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162367 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162372 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162376 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162381 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162386 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162390 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162395 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162400 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162408 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162413 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162418 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162423 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162428 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162433 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162437 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.167427 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162444 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162451 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162458 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162466 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162471 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162476 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162481 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162487 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162493 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162499 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162505 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162510 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162515 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162519 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162524 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162528 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162533 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162537 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162541 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.168038 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162546 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162551 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162555 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162560 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162565 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162569 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162574 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162579 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162583 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162588 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162593 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162600 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162605 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162610 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.162614 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163433 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163444 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163450 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163455 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163460 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.168609 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163465 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163470 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163475 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163480 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163484 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163489 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163494 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163499 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163504 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163508 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163512 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163517 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163522 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163526 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163531 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163536 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163541 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163545 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163550 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.169237 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163556 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163561 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163566 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163571 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163576 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163581 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163585 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163589 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163595 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163600 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163604 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163609 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163614 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163618 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163623 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163628 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163632 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163637 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163642 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163646 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.169841 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163651 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163656 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163660 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163665 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163670 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163676 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163697 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163704 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163709 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163714 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163719 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163724 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163728 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163734 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163739 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163744 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163748 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163753 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163757 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163763 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.170466 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163768 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163773 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163778 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163782 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163787 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163793 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163797 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163802 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163807 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163812 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163817 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163822 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163827 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163832 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163836 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163841 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163845 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163850 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163856 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.171120 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163862 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163867 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163872 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163877 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163881 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163890 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163896 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163901 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163906 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163911 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163916 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163920 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163924 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163929 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163933 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163938 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163943 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163949 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163954 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163959 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.171826 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163963 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163968 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.163973 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164107 5988 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164120 5988 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164129 5988 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164137 5988 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164144 5988 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164150 5988 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164159 5988 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164166 5988 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164172 5988 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164177 5988 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164187 5988 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164193 5988 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164199 5988 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164204 5988 flags.go:64] FLAG: --cgroup-root="" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164212 5988 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164218 5988 flags.go:64] FLAG: --client-ca-file="" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164223 5988 flags.go:64] FLAG: --cloud-provider="external" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164230 5988 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164237 5988 flags.go:64] FLAG: --cluster-domain="" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164243 5988 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164249 5988 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 18:38:45.172670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164254 5988 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164260 5988 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164267 5988 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164273 5988 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164278 5988 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164284 5988 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164290 5988 flags.go:64] FLAG: --contention-profiling="false" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164295 5988 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164300 5988 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164306 5988 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164311 5988 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164318 5988 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164323 5988 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164329 5988 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164334 5988 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164339 5988 flags.go:64] FLAG: --enable-server="true" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164344 5988 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164351 5988 flags.go:64] FLAG: --event-burst="100" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164357 5988 flags.go:64] FLAG: --event-qps="50" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164362 5988 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164367 5988 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164374 5988 flags.go:64] FLAG: --eviction-hard="" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164381 5988 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164387 5988 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164392 5988 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 18:38:45.173550 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164400 5988 flags.go:64] FLAG: --eviction-soft="" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164405 5988 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164411 5988 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164416 5988 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164423 5988 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164429 5988 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164434 5988 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164439 5988 flags.go:64] FLAG: --feature-gates="" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164447 5988 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164452 5988 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164458 5988 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164464 5988 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164469 5988 flags.go:64] FLAG: --healthz-port="10248" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164475 5988 flags.go:64] FLAG: --help="false" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164480 5988 flags.go:64] FLAG: --hostname-override="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164486 5988 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164491 5988 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164496 5988 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164502 5988 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164509 5988 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164513 5988 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164518 5988 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164524 5988 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 18:38:45.174404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164529 5988 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164535 5988 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164540 5988 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164546 5988 flags.go:64] FLAG: --kube-reserved="" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164551 5988 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164559 5988 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164564 5988 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164569 5988 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164575 5988 flags.go:64] FLAG: --lock-file="" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164582 5988 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164587 5988 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164593 5988 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164603 5988 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164609 5988 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164615 5988 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164621 5988 flags.go:64] FLAG: --logging-format="text" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164626 5988 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164632 5988 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164637 5988 flags.go:64] FLAG: --manifest-url="" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164642 5988 flags.go:64] FLAG: --manifest-url-header="" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164651 5988 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164656 5988 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164663 5988 flags.go:64] FLAG: --max-pods="110" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164669 5988 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164674 5988 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 18:38:45.175109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164693 5988 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164699 5988 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164704 5988 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164709 5988 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164715 5988 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164727 5988 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164733 5988 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164738 5988 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164743 5988 flags.go:64] FLAG: --pod-cidr="" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164748 5988 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164753 5988 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164759 5988 flags.go:64] FLAG: --pods-per-core="0" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164767 5988 flags.go:64] FLAG: --port="10250" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164772 5988 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164777 5988 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-028a19785a426fdba" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164785 5988 flags.go:64] FLAG: --qos-reserved="" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164792 5988 flags.go:64] FLAG: --read-only-port="10255" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164798 5988 flags.go:64] FLAG: --register-node="true" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164805 5988 flags.go:64] FLAG: --register-with-taints="" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164812 5988 flags.go:64] FLAG: --registry-burst="10" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164817 5988 flags.go:64] FLAG: --registry-qps="5" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164822 5988 flags.go:64] FLAG: --reserved-cpus="" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164828 5988 flags.go:64] FLAG: --reserved-memory="" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164840 5988 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164846 5988 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164851 5988 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 18:38:45.175857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164857 5988 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164862 5988 flags.go:64] FLAG: --runonce="false" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164867 5988 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164872 5988 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164878 5988 flags.go:64] FLAG: --seccomp-default="false" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164883 5988 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164888 5988 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164894 5988 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164899 5988 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164905 5988 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164910 5988 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164915 5988 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164920 5988 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164925 5988 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164931 5988 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164936 5988 flags.go:64] FLAG: --system-cgroups="" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164941 5988 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164950 5988 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164957 5988 flags.go:64] FLAG: --tls-cert-file="" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164962 5988 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164969 5988 flags.go:64] FLAG: --tls-min-version="" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164974 5988 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164981 5988 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164986 5988 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164991 5988 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 18:38:45.176707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.164997 5988 flags.go:64] FLAG: --v="2" Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.165005 5988 flags.go:64] FLAG: --version="false" Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.165012 5988 flags.go:64] FLAG: --vmodule="" Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.165019 5988 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.165026 5988 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165249 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165256 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165262 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165267 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165272 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165276 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165281 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165286 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165291 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165296 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165301 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165306 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165310 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165315 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165319 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.177548 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165326 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165333 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165339 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165345 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165350 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165358 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165362 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165367 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165372 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165378 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165383 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165388 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165393 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165398 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165402 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165407 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165412 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165418 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165423 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165427 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.178276 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165432 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165437 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165442 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165447 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165452 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165456 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165461 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165465 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165470 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165476 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165481 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165486 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165491 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165496 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165500 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165505 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165510 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165517 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165522 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165527 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.179002 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165532 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165539 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165544 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165548 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165553 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165558 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165563 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165567 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165572 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165602 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165613 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165619 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165624 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165629 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165634 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165639 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165644 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165649 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165653 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.179709 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165658 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165663 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165668 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165672 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165677 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165696 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165701 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165707 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165711 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165716 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165723 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165728 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165733 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165738 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165744 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165749 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165754 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165759 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165763 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165769 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.180322 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165774 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165779 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165784 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165789 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165797 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165804 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165811 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165816 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165822 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165827 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165832 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165837 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.165880 5988 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165937 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165944 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165950 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.181101 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165955 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165960 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165964 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165969 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165975 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165980 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165985 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165989 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165994 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.165999 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166004 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166010 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166015 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166020 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166025 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166030 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166035 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166039 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166044 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166049 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.181612 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166053 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166059 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166064 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166069 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166074 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166079 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166083 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166088 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166093 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166097 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166102 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166107 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166112 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166117 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166122 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166126 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166131 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166135 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166140 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.182209 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166145 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166150 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166155 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166159 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166166 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166170 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166175 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166179 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166185 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166189 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166194 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166199 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166204 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166209 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166213 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166219 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166223 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166229 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166236 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166241 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.182857 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166246 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166253 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166260 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166266 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166272 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166277 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166282 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166286 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166291 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166296 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166300 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166305 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166310 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166314 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166319 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166324 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166331 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166335 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166340 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.183604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166345 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166350 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166354 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166359 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166363 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166370 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166378 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166383 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166389 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166394 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166399 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166404 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166409 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166414 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166419 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166424 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166429 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166433 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166438 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.184281 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166443 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166447 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166452 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166460 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166465 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.166469 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.172036 5988 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.172070 5988 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.172142 5988 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.172161 5988 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172511 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172520 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172525 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172532 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172538 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172543 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172547 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172552 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172556 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.184940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172567 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172571 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172575 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172580 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172584 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172589 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172593 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172597 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172601 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172606 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172610 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172615 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172627 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172633 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172639 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172644 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172648 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172653 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172657 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.185592 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172662 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172714 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172719 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172723 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172728 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172732 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172741 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172745 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172749 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172754 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172758 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172762 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172767 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172771 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172775 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172779 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172784 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172788 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172797 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172802 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.186309 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172806 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172810 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172814 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172819 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172823 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172830 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172835 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172839 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172844 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172848 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172853 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172862 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172867 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172871 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172897 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172904 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172909 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172914 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172918 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.187197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172922 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172927 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172931 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172936 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172945 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172949 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172953 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172957 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172962 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172967 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172971 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172976 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172980 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172984 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172988 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172993 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.172997 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173006 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173010 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.187877 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173015 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173019 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173023 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173028 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173032 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173037 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173815 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173827 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173832 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173839 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173844 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173857 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173860 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173863 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173867 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173872 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173876 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173880 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173885 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.188538 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.173889 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.173930 5988 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174182 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174193 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174199 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174203 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174207 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174211 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174215 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174220 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174224 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174229 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174234 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174239 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174243 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174248 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.189203 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174253 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174257 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174262 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174267 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174271 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174275 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174280 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174285 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174289 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174293 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174298 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174302 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174307 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174311 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174315 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174319 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174322 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174327 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174331 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174335 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.189762 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174338 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174340 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174344 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174347 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174357 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174360 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174363 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174366 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174369 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174372 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174374 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174377 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174380 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174384 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174388 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174392 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174397 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174401 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174404 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174407 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.190508 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174410 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174413 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174415 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174418 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174421 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174424 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174427 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174429 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174432 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174435 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174437 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174440 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174443 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174445 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174448 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174450 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174453 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174461 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174465 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174470 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.191262 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174475 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174478 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174482 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174489 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174494 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174497 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174500 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174503 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174505 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174508 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174511 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174516 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174520 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174524 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174528 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174533 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174538 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174542 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.192049 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174547 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174552 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174557 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174561 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174566 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174571 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174574 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174577 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174580 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174583 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174586 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174589 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174600 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174604 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.174649 5988 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 18:38:45.192672 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174720 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174727 5988 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174730 5988 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174733 5988 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174736 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174739 5988 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174741 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174744 5988 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174748 5988 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174752 5988 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174756 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174760 5988 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174764 5988 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174769 5988 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174772 5988 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174777 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174781 5988 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174786 5988 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174790 5988 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174793 5988 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 18:38:45.193241 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174796 5988 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174800 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174804 5988 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174808 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174813 5988 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174816 5988 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174818 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174821 5988 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174823 5988 feature_gate.go:386] unrecognized feature gate: Example Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174826 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174837 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174841 5988 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174844 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174847 5988 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174850 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174855 5988 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174859 5988 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174863 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174865 5988 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174868 5988 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 18:38:45.194131 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174871 5988 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174874 5988 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174876 5988 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174880 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174883 5988 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174886 5988 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174888 5988 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174891 5988 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174894 5988 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174898 5988 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174902 5988 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174906 5988 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174910 5988 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174914 5988 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174918 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174922 5988 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174926 5988 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174930 5988 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174935 5988 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174938 5988 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 18:38:45.194931 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174941 5988 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174943 5988 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174946 5988 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174958 5988 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174963 5988 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174967 5988 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174971 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174976 5988 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174981 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174986 5988 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174990 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.174999 5988 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175003 5988 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175008 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175013 5988 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175017 5988 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175021 5988 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175026 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175030 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 18:38:45.195766 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175033 5988 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175036 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175039 5988 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175041 5988 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175044 5988 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175046 5988 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175049 5988 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175053 5988 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175057 5988 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175061 5988 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175065 5988 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175070 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175073 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175076 5988 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175081 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175085 5988 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175089 5988 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175102 5988 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175107 5988 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 18:38:45.196544 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175113 5988 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175117 5988 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175121 5988 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175124 5988 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175127 5988 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175133 5988 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175136 5988 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:38:45.175138 5988 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.175224 5988 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.176399 5988 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.176472 5988 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.176702 5988 server.go:1014] "Starting client certificate rotation" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.176799 5988 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.176873 5988 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:32:32 +0000 UTC" deadline="2028-04-21 23:18:31.70819425 +0000 UTC" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.176924 5988 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12844h39m46.53127394s" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.177284 5988 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:38:45.197389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.178630 5988 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 18:38:45.198168 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.180062 5988 log.go:25] "Validated CRI v1 runtime API" Nov 03 18:38:45.198168 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.182315 5988 log.go:25] "Validated CRI v1 image API" Nov 03 18:38:45.198168 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.183198 5988 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 18:38:45.198168 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.185998 5988 fs.go:135] Filesystem UUIDs: map[57f9321f-3745-4e7c-98f7-b24675b7da02:/dev/nvme0n1p4 63570359-2bb8-4093-82f0-7243a515b663:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 18:38:45.198391 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.186022 5988 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d509d8dbb2bff095adbd200728378b133b370108ae2d3c4db787ff0b7aeb85f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d509d8dbb2bff095adbd200728378b133b370108ae2d3c4db787ff0b7aeb85f/userdata/shm major:0 minor:485 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29184f50356ca7962a899b5bca89c9ddd42bda35c351bdae2f28c6c4b4637a81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29184f50356ca7962a899b5bca89c9ddd42bda35c351bdae2f28c6c4b4637a81/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3675f99fba9b96d9fd96cde6b7f691313c771cc53a32ee9a42263af6fa36d01f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3675f99fba9b96d9fd96cde6b7f691313c771cc53a32ee9a42263af6fa36d01f/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d0497e7f8739ba2a8ba132568e7fe4b1f26ac035607c7556b4b9b5aa225b60d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d0497e7f8739ba2a8ba132568e7fe4b1f26ac035607c7556b4b9b5aa225b60d/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68b82928ff6a8143eb8665e0aefaca9162439400823b8bca55b9d3a40c986a5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68b82928ff6a8143eb8665e0aefaca9162439400823b8bca55b9d3a40c986a5a/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478/userdata/shm major:0 minor:463 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/838b57331b9444f84e1f368cfe14a073314d10adf7cf6628368a2dc980ae15e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/838b57331b9444f84e1f368cfe14a073314d10adf7cf6628368a2dc980ae15e9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c3438db408273dcae78047b500d35aee3243e107be76b2bc74dd711f50415c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c3438db408273dcae78047b500d35aee3243e107be76b2bc74dd711f50415c0/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f12979913fac9cba1164846d6b162a29116dcb595d0e8c61a08fb9d95f5de31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f12979913fac9cba1164846d6b162a29116dcb595d0e8c61a08fb9d95f5de31/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9baa51842883a506edc8c8551b6d5b44f1f20b418357b3c0dcb08494ea2e864e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9baa51842883a506edc8c8551b6d5b44f1f20b418357b3c0dcb08494ea2e864e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6841c60d946d0f38ad89adbdb2692618243bda2a07114704a48c91394cf3c32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6841c60d946d0f38ad89adbdb2692618243bda2a07114704a48c91394cf3c32/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b70863381f7a1036a9f5092af393a28a62331aa84e58616260074abb3311cff7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b70863381f7a1036a9f5092af393a28a62331aa84e58616260074abb3311cff7/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf2f025c6739e1e7caf2ed079bc07ac3280927914e54218da1088b9cd3ddd777/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf2f025c6739e1e7caf2ed079bc07ac3280927914e54218da1088b9cd3ddd777/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c268ca90aea9539832c856af0be8630a6c4fec12d6ca9baa4ffcf0bdf0a8f8d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c268ca90aea9539832c856af0be8630a6c4fec12d6ca9baa4ffcf0bdf0a8f8d9/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4e1d14c542291d14732309ff716f11f8f2c281f2e8643cfc51d5369b1d81745/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4e1d14c542291d14732309ff716f11f8f2c281f2e8643cfc51d5369b1d81745/userdata/shm major:0 minor:483 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d271c782a6d0661e6a9a9da46b2e0b775d8de75d0239cfef5e448c90641d7af8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d271c782a6d0661e6a9a9da46b2e0b775d8de75d0239cfef5e448c90641d7af8/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d384cc8598323a5e878607b219947aa1082bbc15298d13d8e7d5f8b22026d53a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d384cc8598323a5e878607b219947aa1082bbc15298d13d8e7d5f8b22026d53a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcf0125738ee90ff43086a2b7df349ef46b378de20c24a42edb70e7bdc5597ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcf0125738ee90ff43086a2b7df349ef46b378de20c24a42edb70e7bdc5597ab/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7ad0f3c2a8f14307e72994f8bde9666335c4e2e6a8ae02fc760677b188605b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7ad0f3c2a8f14307e72994f8bde9666335c4e2e6a8ae02fc760677b188605b0/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0151277ce74115b945de7f56e6b6d52025bcfc709c27643efe0fd1831fb107c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0151277ce74115b945de7f56e6b6d52025bcfc709c27643efe0fd1831fb107c/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f254fc156e78afeb61c659d262483dc80e5d815ca8b1ed3cccb00a90cb518191/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f254fc156e78afeb61c659d262483dc80e5d815ca8b1ed3cccb00a90cb518191/userdata/shm major:0 minor:481 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2aa525220713059f73e53208100102b32390dabc52f8252e668e859f2aa3def/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2aa525220713059f73e53208100102b32390dabc52f8252e668e859f2aa3def/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff3194523636614789fc9d65c78cbd83b2d9b3d623678bfcaab73de797685ad7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff3194523636614789fc9d65c78cbd83b2d9b3d623678bfcaab73de797685ad7/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06841536-c400-4c0f-8d46-5d8e24ba9c6d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/06841536-c400-4c0f-8d46-5d8e24ba9c6d/volumes/kubernetes.io~secret/agent-certs major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08020da4-3f4c-4eef-8e5f-126abd177058/volumes/kubernetes.io~projected/kube-api-access-b4mqm:{mountpoint:/var/lib/kubelet/pods/08020da4-3f4c-4eef-8e5f-126abd177058/volumes/kubernetes.io~projected/kube-api-access-b4mqm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08020da4-3f4c-4eef-8e5f-126abd177058/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/08020da4-3f4c-4eef-8e5f-126abd177058/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f0cd7d8-5868-443d-88a4-8d2e464631e9/volumes/kubernetes.io~projected/kube-api-access-h9fjt:{mountpoint:/var/lib/kubelet/pods/0f0cd7d8-5868-443d-88a4-8d2e464631e9/volumes/kubernetes.io~projected/kube-api-access-h9fjt major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14054e49-309c-46d4-ba51-5a2ad669f1a0/volumes/kubernetes.io~projected/kube-api-access-vltb7:{mountpoint:/var/lib/kubelet/pods/14054e49-309c-46d4-ba51-5a2ad669f1a0/volumes/kubernetes.io~projected/kube-api-access-vltb7 major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f140b25-ea89-4670-8e35-047a4bd190ac/volumes/kubernetes.io~projected/kube-api-access-qfxjv:{mountpoint:/var/lib/kubelet/pods/1f140b25-ea89-4670-8e35-047a4bd190ac/volumes/kubernetes.io~projected/kube-api-access-qfxjv major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f140b25-ea89-4670-8e35-047a4bd190ac/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1f140b25-ea89-4670-8e35-047a4bd190ac/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~projected/kube-api-access-gwl46:{mountpoint:/var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~projected/kube-api-access-gwl46 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c91d4f5-0557-4eda-abb2-9f49e237d4b5/volumes/kubernetes.io~projected/kube-api-access-mhtjx:{mountpoint:/var/lib/kubelet/pods/2c91d4f5-0557-4eda-abb2-9f49e237d4b5/volumes/kubernetes.io~projected/kube-api-access-mhtjx major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~empty-dir/tmp major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~projected/kube-api-access-szmjb:{mountpoint:/var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~projected/kube-api-access-szmjb major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~projected/kube-api-access-fvzv4:{mountpoint:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~projected/kube-api-access-fvzv4 major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~projected/kube-api-access-pf78j:{mountpoint:/var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~projected/kube-api-access-pf78j major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4322a265-eccb-4e0b-89e3-8844e35f3b6e/volumes/kubernetes.io~projected/kube-api-access-p5rfm:{mountpoint:/var/lib/kubelet/pods/4322a265-eccb-4e0b-89e3-8844e35f3b6e/volumes/kubernetes.io~projected/kube-api-access-p5rfm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4322a265-eccb-4e0b-89e3-8844e35f3b6e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4322a265-eccb-4e0b-89e3-8844e35f3b6e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:478 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce/volumes/kubernetes.io~projected/kube-api-access-psvqf:{mountpoint:/var/lib/kubelet/pods/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce/volumes/kubernetes.io~projected/kube-api-access-psvqf major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volumes/kubernetes.io~projected/kube-api-access-5ktm4:{mountpoint:/var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volumes/kubernetes.io~projected/kube-api-access-5ktm4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d25cdfb-758e-47fa-adbc-8f6953a7be99/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4d25cdfb-758e-47fa-adbc-8f6953a7be99/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/736cce59-1d70-4b20-975d-58d36563698f/volumes/kubernetes.io~projected/kube-api-access-2kmnm:{mountpoint:/var/lib/kubelet/pods/736cce59-1d70-4b20-975d-58d36563698f/volumes/kubernetes.io~projected/kube-api-access-2kmnm major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/config-reloader/2 major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/prometheus/15 major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~empty-dir/config-out major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~projected/kube-api-access-smrrx:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~projected/kube-api-access-smrrx major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~projected/tls-assets major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/config major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/web-config major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~projected/kube-api-access-jgxcg:{mountpoint:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~projected/kube-api-access-jgxcg major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7601adc3-f067-48c5-b556-58b607361367/volumes/kubernetes.io~projected/kube-api-access-2nxsq:{mountpoint:/var/lib/kubelet/pods/7601adc3-f067-48c5-b556-58b607361367/volumes/kubernetes.io~projected/kube-api-access-2nxsq major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d/volumes/kubernetes.io~projected/kube-api-access-bfx8d:{mountpoint:/var/lib/kubelet/pods/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d/volumes/kubernetes.io~projected/kube-api-access-bfx8d major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~projected/kube-api-access-hlr8v:{mountpoint:/var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~projected/kube-api-access-hlr8v major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7933f6c-3101-4ee6-af60-ecec7ac89f03/volumes/kubernetes.io~projected/kube-api-access-sfq9n:{mountpoint:/var/lib/kubelet/pods/a7933f6c-3101-4ee6-af60-ecec7ac89f03/volumes/kubernetes.io~projected/kube-api-access-sfq9n major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aedaa821-137b-4371-9a23-c4a2890cd484/volumes/kubernetes.io~projected/kube-api-access-c5zw8:{mountpoint:/var/lib/kubelet/pods/aedaa821-137b-4371-9a23-c4a2890cd484/volumes/kubernetes.io~projected/kube-api-access-c5zw8 major:0 minor:480 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~projected/kube-api-access-wcwdr:{mountpoint:/var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~projected/kube-api-access-wcwdr major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~projected/kube-api-access-6snlf:{mountpoint:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~projected/kube-api-access-6snlf major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/alertmanager/9 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/config-reloader/6 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~empty-dir/config-out major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~projected/kube-api-access-bxgs4:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~projected/kube-api-access-bxgs4 major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~projected/tls-assets major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/config-volume major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/web-config major:0 minor:322 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6332c08c8a11f9029e0ed32ce5deefbf1c8d6d7e2cfdee2737897504982d5bcb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/801989b09c55ccc1a5bee607953c01c9c296a02f3495a7add460e7715f8d98cd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/571fa2c2e38daeebdaa5c50e3a8a258867f7408637d581e145c4c043f3f6af66/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/b007768464da7a9ce0e713f260825f4d8dbbed935e8309172aebda74dc84e9b4/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/d816638502777642013e622ce2057a2dffb250a19cc52336016e9b84c1bbee9a/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a6fd0bce3495cb03a3807aa60137db9e4b703df859cdd53d9408a9d91a9db110/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9e67cc915c8363b9b84e71ebfc73bf03e3efc5ee2196d403e3d2665df328f176/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b2b2b23c175b0bb82ddcc75ef92c77e96d536226c73bbb1a3848f089479aad77/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f27543ce85cab2c0ecc929b1e0d7e357770d07ee563db691de6d31ebc4761a2b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ed241e6b03071312f07e658c7fdd9f6c6a9521ced7d719027b4cdb2af0929842/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/19650849d30e6d79b7f9122f25a8d76ba0373cbf4716a9c0108d110b5a431a6f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/049c45072c17226b010b6e9cb09f3f81543ad2b8e65e9f555d48ec149bae19fe/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8a4eaad9eca9987fac9f6a416db03805f63f30080ce49a9673ba3596143f1eea/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/77b56fc975a9efd2318a285969fd5f8cdc6ceb861ade3f4d012ed7aec65d11b3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4d85418b30b73be6da99d4f8e5d2ada1e97c31c1cb75f28590ea8e592cdba17a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ed48c8aa7a12864a114c08315fc80e216135d661cbcb56d703211818cdeb36d6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0b0fa865de56861ebec7816484dd55f6aec4286139ad4e02e63e039362a45a3b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/fbd10b69631a95efcec812702637d0677e576b26c692f0fce9feb99444c88ab3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ae3dfedda502515754ac654fc4ebb3a317a3d5895cd60a38b6f4ffe2ed66cfef/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/dd3d184e450eec2912ef836ba6a93bf7f1b622ba3913117b2b1d96c474553a76/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c82fd2874a7c2a4f1e85f9daf176ef173a003c07b69645e567fd2e75668c9e77/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d298fd8114666dc7bbc36c9abf7ea314711ee9fdc55df55b17588a84d87bf32b/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f432e8dae6193f4a969142ca8e63474c37a20c8f932f82396b5cbfb011228f4d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6850a69df6ae0ee3b504a3674fcda6a8b68ba9917c5d14c48fbe0fe344d85016/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/3104b95d127381b7789ca14a0cbd39aaf36c5f86a70f79673c42423c2c67ed8a/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/e7916d589497b8a4fbd7694a0a94e09a79dec967c845d39d7a6e04bc364bb2ac/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/346dff95802ce092ee133d5000f77f76792abbf5700e83e005b936c2997476b2/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/d88cf32aa48a5958c7a92038b3439d70639035d66bbc2fb67b2762220cb56821/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/91dfd77739109e85b6e209ac49966e5464601a7563801be4e3361450e0c4c0b0/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/3d96e141afabd110c9c0fb475561b66e059052140546dc2a92cebaa71ccf9752/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d7914fcc7b25639b7bb260b4087779b47c56fc74aed3dce670a44895181995aa/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/e6b0606ad12bc3bdfd500c187ba869a3dfb574ab4df0e4a70558e27a596d3865/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c0ca462d0444d2214476f305906be329370833d16a0a94f596c3cb83fadd32e9/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/d41c6ae8e2ed87d6c70e9b20dfe342c858a276b3748a59a7ddaea0cd3dcb9abd/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/8672db32587be2dcd27cd148d519b6855c1ceea7ea6988a8a87385b0ab55a19a/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/d66beb3e0b696cafebc1b0d7244679e60e7f1148a591bb58f90417de85fba1b8/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/9f29cc8c29dd8424da2b1d584acc3877472a8becc693100dfe8ff12df85d066e/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/884622fb3750057117c26e1c927077801cd719ca2a0290ca7ddadf1402d64210/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/410719d3c8422e52ce4d92158889b4db4260a22502f310eebabf94145652c646/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/fcf73966111be9fe637fd4aa81aa824837b1791c38e14464f57bcbe4c12dc9c9/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/c3ad98c9ea0ecaaa3acdb41c125f9c0e6bfa3be6840f291705fbf60262bee963/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/c27455e20a99f341e17ad1da20f5557827d15eff4d460376a7d5254fde4031fb/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/d6fd3246c29f25481f2f7ae623ae307ca4d66b70f16a51a7def7df80e546f80b/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/454050463a640cbd72df9c582833d378814841b4bff9a539876a9a01f20ba824/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/163c71b182ed0a9ba81f36b8a865afe485a6c195ff44103bea3adc2063150387/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4a4e21f007655d0ffd9c20e04075f24a2229331ea4d68c7046f3e50377886277/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/f878eac8e025ef2b0815c556d3766a5d5025d4abe399c7d5f9b1f019d316d461/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/357042fd016ec2d19c39842c86e754164fb8ca1125a6157551eb58132a38fada/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/eaee2f12cf63cdb19f861aadeddadf2180c8dea6f6f3c2b5ea3667ab082f4fa5/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/3762440969d4cbe422ae5c1ddebeb8543ff0fe3533f40369db137636d38620fe/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/00c73f8e4b7f951ce3bcfd479d30ee75e8e89bc94e4f9b9fb3bcf8083825b130/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/78e1b4e54f8225b2397af1879cf200eec818fdd5677c89ddaf039e9adb987a15/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/2343e08cee9554bfe5c3204ee294ad1aba261aa45dc52c948e969f0044f4f082/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/6fe343d7a48258cb9f52874e3d93b8167a5480508f8b96c9fc272eb4f295048d/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/799c40bb0d1021b46715f69e8d50400e70fd53c77f641e2a1e7ee1c86a3bca47/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/b200a9a50621f412cbb15ab457ac4f741c32a3edd84bb89463b298d38d148fec/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/4593bebea19fc4edf0237cb372fe333d4ca4eb9260122db11a39acd5a26e055e/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/79be86b449ca22859d581fada033d0160a40be5aa17a71ad5e4d9bc3c70511e5/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/589c0b3407ae7e478690f5a01503a40812ffaaa7ab20454ae2ae40d77517d7a0/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/10022766e7bb963b6b5f531487e1f96d6160a34cddf3e2d5feb76621e04200a7/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/7e26b06b389decfcff7d5825aafb0871e0806c49cfa590ff9f3553ee3a6058a8/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/025447fd2b106df567964361abbbfd5b5f6b20df8795b6b05de068aeaca66adf/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/ff8c5e3330f9f95f7d8097e73da984a0943bcfa57af67b70e1eb9c37e45de2ad/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/76e3c3be24efa40fb01ef50d006fef3f0976e8ac5c07cbb6ff31f6ddf54d2e82/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/15013f6c32ff68201f56585c44d0d6a788b7268f2b62660e1de0a308586e99f7/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/942d07aa85d66a23a93a17f591644a1b110f7868cb5eeb4ab38c6103d62be1b4/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/41f6f0b579bbb9a017f2f6d996b9a1c0d9b6587cf810a5d64bd80ffbcbc9f93b/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/a1013811ea9ffe527001537b5886a9c11d021a1eb58796335864576cd0308b27/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/4f7711b3618ae9cc3287dfbdeda72c1248d02a6145c18587de5895bb887ed0b2/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/105944a4ff82c6e59557c91d60408592982a8f4c7f88369c8f29e59c096134ab/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/6d582e39ae759741c588e102cc1dc610d08f398e8cbc1aa7fcbc731111fa1fb2/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/1bc5589cb491f97ad9b0c3c0fa43c85146e551e827c3bd72c0667a3cd1f37cb5/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/f72a52f0fff7330b43a656ec28b9e52f584ac66f86dfe2dbbda684bbf7813668/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/4b51145e8069866cbb1627abe3fd33c2c41912bbe7c7c22233d8629e380fd520/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-517:{mountpoint:/var/lib/containers/storage/overlay/dc877a0fff64765ff13b3fce93aec24eecbc1b1531adb52d23b5cef45262739e/merged major:0 minor:517 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/bd3d3cd265ef27468aa84ae5701494444e01d52a650da45e396b80c7bf3a42a0/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/ee7139901b120e82cba866cb4a564d0cdedc7dd5084a0dfacee506dced45d40d/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-532:{mountpoint:/var/lib/containers/storage/overlay/8bec9120749d3e33467e6398bdd2aa4c431f53840ad5fbf2c3170d9e9db64b23/merged major:0 minor:532 fsType:overlay blockSize:0} overlay_0-537:{mountpoint:/var/lib/containers/storage/overlay/5c2b8c7f4469e4899bbe7f35680d3639d87287140e210b6dcc3e075526ec1638/merged major:0 minor:537 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/87e1dbbbe6a39e9d8a415dd170091189225420f0faa5ce2826d11dc67a49d65d/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/b5040a89d03bbf7e48d55d47ce269a692c283a17655880845ce1a422548dd01d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b54f112bd205fee2ce803bbb7de2622cd986e1dceb1b98653f51875c2b47b3db/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a7fdc8a594f575f471c962f26a5bedb166f901c460f4c5570f152a274c040b8b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/10c0f673da6b7e4fc30aac8c12b81189fa4bd90bb2e9e27eee1af6792c2b2d0c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/2d08341b2043a559a20faaf605ddd343711712db2833d134bdabe971e6e758d3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7d0e3c813fe2da70d5c02899d94c151e30b04f93006e64e700ca7a7e0a6a6312/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3cea334b024b8fbd1fcf399bd40a816991e1a0c3921e505a1d1ecce8680570e0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/efa7451e2e7b9755d0a69d004713208b7a1e87ff89dded31120393762f0a5700/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/94b9fcccd6eb6cf35422c041c635720574e5aebcb8938f99c8719de9018c4237/merged major:0 minor:97 fsType:overlay blockSize:0}] Nov 03 18:38:45.198915 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.197902 5988 manager.go:217] Machine: {Timestamp:2026-11-03 18:38:45.197337426 +0000 UTC m=+0.078753529 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099738 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c849cf7c1889eb667fb742e16b718 SystemUUID:ec2c849c-f7c1-889e-b667-fb742e16b718 BootID:418ba0f6-fd00-4fee-abc4-45df2f5d32e4 Filesystems:[{Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff3194523636614789fc9d65c78cbd83b2d9b3d623678bfcaab73de797685ad7/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7601adc3-f067-48c5-b556-58b607361367/volumes/kubernetes.io~projected/kube-api-access-2nxsq DeviceMajor:0 DeviceMinor:325 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0f0cd7d8-5868-443d-88a4-8d2e464631e9/volumes/kubernetes.io~projected/kube-api-access-h9fjt DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:362 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c268ca90aea9539832c856af0be8630a6c4fec12d6ca9baa4ffcf0bdf0a8f8d9/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:446 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~projected/kube-api-access-smrrx DeviceMajor:0 DeviceMinor:454 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d/volumes/kubernetes.io~projected/kube-api-access-bfx8d DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0151277ce74115b945de7f56e6b6d52025bcfc709c27643efe0fd1831fb107c/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~projected/kube-api-access-gwl46 DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:371 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~projected/kube-api-access-pf78j DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:447 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4e1d14c542291d14732309ff716f11f8f2c281f2e8643cfc51d5369b1d81745/userdata/shm DeviceMajor:0 DeviceMinor:483 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b70863381f7a1036a9f5092af393a28a62331aa84e58616260074abb3311cff7/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:319 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volumes/kubernetes.io~projected/kube-api-access-5ktm4 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/42604250-348d-4471-8d8a-d377e8986eaf/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:450 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/838b57331b9444f84e1f368cfe14a073314d10adf7cf6628368a2dc980ae15e9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4322a265-eccb-4e0b-89e3-8844e35f3b6e/volumes/kubernetes.io~projected/kube-api-access-p5rfm DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d384cc8598323a5e878607b219947aa1082bbc15298d13d8e7d5f8b22026d53a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478/userdata/shm DeviceMajor:0 DeviceMinor:463 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68b82928ff6a8143eb8665e0aefaca9162439400823b8bca55b9d3a40c986a5a/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcf0125738ee90ff43086a2b7df349ef46b378de20c24a42edb70e7bdc5597ab/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce/volumes/kubernetes.io~projected/kube-api-access-psvqf DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d271c782a6d0661e6a9a9da46b2e0b775d8de75d0239cfef5e448c90641d7af8/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:274 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~projected/kube-api-access-bxgs4 DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6841c60d946d0f38ad89adbdb2692618243bda2a07114704a48c91394cf3c32/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2aa525220713059f73e53208100102b32390dabc52f8252e668e859f2aa3def/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:448 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c91d4f5-0557-4eda-abb2-9f49e237d4b5/volumes/kubernetes.io~projected/kube-api-access-mhtjx DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:363 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-532 DeviceMajor:0 DeviceMinor:532 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-537 DeviceMajor:0 DeviceMinor:537 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/06841536-c400-4c0f-8d46-5d8e24ba9c6d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:320 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:372 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f3c563d-6155-43a3-bff6-5e95b9cee38d/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:333 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:449 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-517 DeviceMajor:0 DeviceMinor:517 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9baa51842883a506edc8c8551b6d5b44f1f20b418357b3c0dcb08494ea2e864e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f254fc156e78afeb61c659d262483dc80e5d815ca8b1ed3cccb00a90cb518191/userdata/shm DeviceMajor:0 DeviceMinor:481 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:369 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~projected/kube-api-access-fvzv4 DeviceMajor:0 DeviceMinor:373 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~projected/kube-api-access-szmjb DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:445 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:449 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14054e49-309c-46d4-ba51-5a2ad669f1a0/volumes/kubernetes.io~projected/kube-api-access-vltb7 DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f12979913fac9cba1164846d6b162a29116dcb595d0e8c61a08fb9d95f5de31/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:331 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:444 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:449 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~projected/kube-api-access-wcwdr DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d25cdfb-758e-47fa-adbc-8f6953a7be99/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:479 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:449 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46f7e70a-abc0-49c1-a7a5-15ef4ff91925/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/08020da4-3f4c-4eef-8e5f-126abd177058/volumes/kubernetes.io~projected/kube-api-access-b4mqm DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf2f025c6739e1e7caf2ed079bc07ac3280927914e54218da1088b9cd3ddd777/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~projected/kube-api-access-6snlf DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/736cce59-1d70-4b20-975d-58d36563698f/volumes/kubernetes.io~projected/kube-api-access-2kmnm DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1f140b25-ea89-4670-8e35-047a4bd190ac/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:361 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d64f9eb-a543-4518-8d1d-0a27cac5a88a/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:370 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7ad0f3c2a8f14307e72994f8bde9666335c4e2e6a8ae02fc760677b188605b0/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aedaa821-137b-4371-9a23-c4a2890cd484/volumes/kubernetes.io~projected/kube-api-access-c5zw8 DeviceMajor:0 DeviceMinor:480 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/38dd707d-5338-4e99-a3b4-49380731aebb/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d0497e7f8739ba2a8ba132568e7fe4b1f26ac035607c7556b4b9b5aa225b60d/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9b32f9a9-d324-4501-9c23-4e93b00863aa/volumes/kubernetes.io~projected/kube-api-access-hlr8v DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b07cbb8d-1cfb-425b-8de7-e8ae88befa55/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:273 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74f5b2ca-aa33-444a-bd6f-56df2278d099/volumes/kubernetes.io~projected/kube-api-access-jgxcg DeviceMajor:0 DeviceMinor:364 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d509d8dbb2bff095adbd200728378b133b370108ae2d3c4db787ff0b7aeb85f/userdata/shm DeviceMajor:0 DeviceMinor:485 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:437 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f140b25-ea89-4670-8e35-047a4bd190ac/volumes/kubernetes.io~projected/kube-api-access-qfxjv DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c026965c-43dc-41e4-a73a-90ba867305a5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:332 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a7933f6c-3101-4ee6-af60-ecec7ac89f03/volumes/kubernetes.io~projected/kube-api-access-sfq9n DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/08020da4-3f4c-4eef-8e5f-126abd177058/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:322 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:442 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:443 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29184f50356ca7962a899b5bca89c9ddd42bda35c351bdae2f28c6c4b4637a81/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:324 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:441 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4322a265-eccb-4e0b-89e3-8844e35f3b6e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:478 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c3438db408273dcae78047b500d35aee3243e107be76b2bc74dd711f50415c0/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3675f99fba9b96d9fd96cde6b7f691313c771cc53a32ee9a42263af6fa36d01f/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0d509d8dbb2bff0 MacAddress:2e:4d:74:e1:76:ab Speed:10000 Mtu:8901} {Name:3675f99fba9b96d MacAddress:72:43:9d:e1:c0:8f Speed:10000 Mtu:8901} {Name:6001ab64e54c562 MacAddress:96:28:66:21:05:b1 Speed:10000 Mtu:8901} {Name:68b82928ff6a814 MacAddress:0e:56:9b:7c:72:87 Speed:10000 Mtu:8901} {Name:68f415841edd579 MacAddress:92:d7:fc:42:fc:f4 Speed:10000 Mtu:8901} {Name:8c3438db408273d MacAddress:6e:9c:f9:c7:94:5f Speed:10000 Mtu:8901} {Name:a6841c60d946d0f MacAddress:56:b3:06:1b:9f:42 Speed:10000 Mtu:8901} {Name:bf2f025c6739e1e MacAddress:2a:5b:2a:79:d6:47 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:dd:3e:19:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:5e:5b:13:ab:22:97 Speed:0 Mtu:8901} {Name:c268ca90aea9539 MacAddress:3a:b6:07:23:d4:cd Speed:10000 Mtu:8901} {Name:c4e1d14c542291d MacAddress:ca:ab:5a:10:aa:d8 Speed:10000 Mtu:8901} {Name:d271c782a6d0661 MacAddress:56:12:93:67:24:21 Speed:10000 Mtu:8901} {Name:e7ad0f3c2a8f143 MacAddress:f6:63:e7:e3:2b:da Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:dd:3e:19:69 Speed:0 Mtu:9001} {Name:f254fc156e78afe MacAddress:7e:91:36:a5:78:1c Speed:10000 Mtu:8901} {Name:f2aa52522071305 MacAddress:ba:67:a6:7c:aa:64 Speed:10000 Mtu:8901} {Name:ff3194523636614 MacAddress:46:db:28:b8:9c:b5 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1a:72:2c:3b:f2:06 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ca:68:df:95:a9:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 18:38:45.198915 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.198450 5988 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 18:38:45.198915 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.198501 5988 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 18:38:45.198915 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.198873 5988 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 18:38:45.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.198903 5988 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-161.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 18:38:45.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199068 5988 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 18:38:45.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199076 5988 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 18:38:45.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199087 5988 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:38:45.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199106 5988 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 18:38:45.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199207 5988 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 18:38:45.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199429 5988 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 18:38:45.199885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199604 5988 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 18:38:45.199885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199642 5988 kubelet.go:493] "Attempting to sync node with API server" Nov 03 18:38:45.199885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199655 5988 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 18:38:45.199885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199674 5988 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 18:38:45.199885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199714 5988 kubelet.go:405] "Adding apiserver pod source" Nov 03 18:38:45.199885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.199723 5988 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 18:38:45.200834 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.200806 5988 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:38:45.200834 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.200819 5988 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:38:45.201408 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.201361 5988 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 18:38:45.219164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219139 5988 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 18:38:45.219164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219162 5988 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 18:38:45.219366 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219302 5988 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 18:38:45.219563 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219548 5988 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 18:38:45.219623 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219597 5988 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 18:38:45.219757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219744 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219766 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219775 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219784 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219793 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219799 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219807 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219814 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219828 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 18:38:45.219833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219837 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 18:38:45.220198 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219855 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 18:38:45.220198 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219868 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 18:38:45.220198 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219906 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 18:38:45.220198 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.219916 5988 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 18:38:45.224293 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.224272 5988 server.go:1263] "Started kubelet" Nov 03 18:38:45.224508 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.224477 5988 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 18:38:45.225021 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.224638 5988 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 18:38:45.225021 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.224731 5988 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 18:38:45.224842 ip-10-0-139-161 systemd[1]: Started Kubernetes Kubelet. Nov 03 18:38:45.225347 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.225030 5988 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 18:38:45.226345 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.225924 5988 server.go:317] "Adding debug handlers to kubelet server" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.235545 5988 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.235566 5988 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.235677 5988 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.235712 5988 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.235739 5988 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:32:33 +0000 UTC" deadline="2028-03-29 07:51:23.363582862 +0000 UTC" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.235756 5988 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12277h12m38.127828401s" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.235807 5988 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.236073 5988 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.236776 5988 factory.go:55] Registering systemd factory Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.236797 5988 factory.go:223] Registration of the systemd container factory successfully Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.237232 5988 factory.go:153] Registering CRI-O factory Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.237250 5988 factory.go:223] Registration of the crio container factory successfully Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.237307 5988 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.237329 5988 factory.go:103] Registering Raw factory Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.237344 5988 manager.go:1199] Started watching for new ooms in manager Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.237740 5988 manager.go:319] Starting recovery of all containers Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239522 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239564 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" volumeName="kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239579 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239590 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239598 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/projected/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-kube-api-access-fvzv4" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239606 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" volumeName="kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-env-overrides" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239614 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239622 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239645 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239654 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06841536-c400-4c0f-8d46-5d8e24ba9c6d" volumeName="kubernetes.io/secret/06841536-c400-4c0f-8d46-5d8e24ba9c6d-agent-certs" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239662 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42604250-348d-4471-8d8a-d377e8986eaf" volumeName="kubernetes.io/configmap/42604250-348d-4471-8d8a-d377e8986eaf-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239669 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" volumeName="kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-tls" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239676 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239700 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239714 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239727 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3c563d-6155-43a3-bff6-5e95b9cee38d" volumeName="kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.239721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239737 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239794 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06841536-c400-4c0f-8d46-5d8e24ba9c6d" volumeName="kubernetes.io/configmap/06841536-c400-4c0f-8d46-5d8e24ba9c6d-konnectivity-ca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239805 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08020da4-3f4c-4eef-8e5f-126abd177058" volumeName="kubernetes.io/configmap/08020da4-3f4c-4eef-8e5f-126abd177058-config-volume" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239817 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08020da4-3f4c-4eef-8e5f-126abd177058" volumeName="kubernetes.io/projected/08020da4-3f4c-4eef-8e5f-126abd177058-kube-api-access-b4mqm" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239828 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239839 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239860 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239872 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239883 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4322a265-eccb-4e0b-89e3-8844e35f3b6e" volumeName="kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239894 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d25cdfb-758e-47fa-adbc-8f6953a7be99" volumeName="kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239904 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239918 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0cd7d8-5868-443d-88a4-8d2e464631e9" volumeName="kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-binary-copy" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239954 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239967 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f140b25-ea89-4670-8e35-047a4bd190ac" volumeName="kubernetes.io/projected/1f140b25-ea89-4670-8e35-047a4bd190ac-kube-api-access-qfxjv" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239979 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42604250-348d-4471-8d8a-d377e8986eaf" volumeName="kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.239991 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43875fcc-00e2-40f4-b610-f4c8aa6ac7ce" volumeName="kubernetes.io/empty-dir/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-data-volume" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240002 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7601adc3-f067-48c5-b556-58b607361367" volumeName="kubernetes.io/empty-dir/7601adc3-f067-48c5-b556-58b607361367-tmp" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240016 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08020da4-3f4c-4eef-8e5f-126abd177058" volumeName="kubernetes.io/empty-dir/08020da4-3f4c-4eef-8e5f-126abd177058-tmp-dir" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240026 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240039 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240050 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f5b2ca-aa33-444a-bd6f-56df2278d099" volumeName="kubernetes.io/projected/74f5b2ca-aa33-444a-bd6f-56df2278d099-kube-api-access-jgxcg" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240063 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/configmap/c026965c-43dc-41e4-a73a-90ba867305a5-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240087 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240099 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240110 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-client-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240123 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43875fcc-00e2-40f4-b610-f4c8aa6ac7ce" volumeName="kubernetes.io/projected/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-api-access-psvqf" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240162 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedaa821-137b-4371-9a23-c4a2890cd484" volumeName="kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240177 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240190 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240202 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240214 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f5b2ca-aa33-444a-bd6f-56df2278d099" volumeName="kubernetes.io/empty-dir/74f5b2ca-aa33-444a-bd6f-56df2278d099-audit-log" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240229 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f5b2ca-aa33-444a-bd6f-56df2278d099" volumeName="kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-client-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240243 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42604250-348d-4471-8d8a-d377e8986eaf" volumeName="kubernetes.io/projected/42604250-348d-4471-8d8a-d377e8986eaf-kube-api-access-pf78j" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240256 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" volumeName="kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240269 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240282 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38dd707d-5338-4e99-a3b4-49380731aebb" volumeName="kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-tmp" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240296 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f140b25-ea89-4670-8e35-047a4bd190ac" volumeName="kubernetes.io/secret/1f140b25-ea89-4670-8e35-047a4bd190ac-cert" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240322 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240334 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d" volumeName="kubernetes.io/projected/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-kube-api-access-bfx8d" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240347 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0cd7d8-5868-443d-88a4-8d2e464631e9" volumeName="kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240370 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3c563d-6155-43a3-bff6-5e95b9cee38d" volumeName="kubernetes.io/empty-dir/1f3c563d-6155-43a3-bff6-5e95b9cee38d-volume-directive-shadow" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240383 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d" volumeName="kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-daemon-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240396 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240409 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240421 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42604250-348d-4471-8d8a-d377e8986eaf" volumeName="kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240433 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7601adc3-f067-48c5-b556-58b607361367" volumeName="kubernetes.io/projected/7601adc3-f067-48c5-b556-58b607361367-kube-api-access-2nxsq" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240444 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d" volumeName="kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cni-binary-copy" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240457 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736cce59-1d70-4b20-975d-58d36563698f" volumeName="kubernetes.io/projected/736cce59-1d70-4b20-975d-58d36563698f-kube-api-access-2kmnm" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240469 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240484 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/projected/c026965c-43dc-41e4-a73a-90ba867305a5-kube-api-access-6snlf" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240504 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240516 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c91d4f5-0557-4eda-abb2-9f49e237d4b5" volumeName="kubernetes.io/empty-dir/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-tmp-dir" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240529 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240543 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b32f9a9-d324-4501-9c23-4e93b00863aa" volumeName="kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240555 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b32f9a9-d324-4501-9c23-4e93b00863aa" volumeName="kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240569 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240581 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240594 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240606 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c91d4f5-0557-4eda-abb2-9f49e237d4b5" volumeName="kubernetes.io/projected/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-kube-api-access-mhtjx" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240618 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0cd7d8-5868-443d-88a4-8d2e464631e9" volumeName="kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240632 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3c563d-6155-43a3-bff6-5e95b9cee38d" volumeName="kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240645 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240657 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f5b2ca-aa33-444a-bd6f-56df2278d099" volumeName="kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240673 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" volumeName="kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240700 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14054e49-309c-46d4-ba51-5a2ad669f1a0" volumeName="kubernetes.io/configmap/14054e49-309c-46d4-ba51-5a2ad669f1a0-serviceca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240714 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b32f9a9-d324-4501-9c23-4e93b00863aa" volumeName="kubernetes.io/projected/9b32f9a9-d324-4501-9c23-4e93b00863aa-kube-api-access-hlr8v" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240727 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" volumeName="kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240740 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4322a265-eccb-4e0b-89e3-8844e35f3b6e" volumeName="kubernetes.io/projected/4322a265-eccb-4e0b-89e3-8844e35f3b6e-kube-api-access-p5rfm" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240752 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43875fcc-00e2-40f4-b610-f4c8aa6ac7ce" volumeName="kubernetes.io/configmap/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240765 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43875fcc-00e2-40f4-b610-f4c8aa6ac7ce" volumeName="kubernetes.io/secret/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240777 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240790 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38dd707d-5338-4e99-a3b4-49380731aebb" volumeName="kubernetes.io/projected/38dd707d-5338-4e99-a3b4-49380731aebb-kube-api-access-szmjb" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240803 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240816 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" volumeName="kubernetes.io/projected/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-kube-api-access-wcwdr" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240830 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f5b2ca-aa33-444a-bd6f-56df2278d099" volumeName="kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240842 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f5b2ca-aa33-444a-bd6f-56df2278d099" volumeName="kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240854 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240868 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b32f9a9-d324-4501-9c23-4e93b00863aa" volumeName="kubernetes.io/configmap/9b32f9a9-d324-4501-9c23-4e93b00863aa-metrics-client-ca" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240880 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/empty-dir/c026965c-43dc-41e4-a73a-90ba867305a5-tmp" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240896 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240909 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14054e49-309c-46d4-ba51-5a2ad669f1a0" volumeName="kubernetes.io/projected/14054e49-309c-46d4-ba51-5a2ad669f1a0-kube-api-access-vltb7" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240923 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3c563d-6155-43a3-bff6-5e95b9cee38d" volumeName="kubernetes.io/projected/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-api-access-gwl46" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240935 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" volumeName="kubernetes.io/empty-dir/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-textfile" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240947 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0cd7d8-5868-443d-88a4-8d2e464631e9" volumeName="kubernetes.io/projected/0f0cd7d8-5868-443d-88a4-8d2e464631e9-kube-api-access-h9fjt" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240960 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3c563d-6155-43a3-bff6-5e95b9cee38d" volumeName="kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240974 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240986 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74f5b2ca-aa33-444a-bd6f-56df2278d099" volumeName="kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.240998 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c026965c-43dc-41e4-a73a-90ba867305a5" volumeName="kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-grpc-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241013 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" volumeName="kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241026 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d64f9eb-a543-4518-8d1d-0a27cac5a88a" volumeName="kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-federate-client-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241039 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" volumeName="kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-script-lib" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241053 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f3c563d-6155-43a3-bff6-5e95b9cee38d" volumeName="kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241065 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" volumeName="kubernetes.io/projected/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-kube-api-access-5ktm4" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241077 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="736cce59-1d70-4b20-975d-58d36563698f" volumeName="kubernetes.io/configmap/736cce59-1d70-4b20-975d-58d36563698f-iptables-alerter-script" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241090 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46f7e70a-abc0-49c1-a7a5-15ef4ff91925" volumeName="kubernetes.io/secret/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241103 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08020da4-3f4c-4eef-8e5f-126abd177058" volumeName="kubernetes.io/secret/08020da4-3f4c-4eef-8e5f-126abd177058-metrics-tls" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241135 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38dd707d-5338-4e99-a3b4-49380731aebb" volumeName="kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-etc-tuned" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241150 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241162 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241174 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74be8f9c-362e-4589-b78f-d72f46a83ebc" volumeName="kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241186 5988 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7933f6c-3101-4ee6-af60-ecec7ac89f03" volumeName="kubernetes.io/projected/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kube-api-access-sfq9n" seLinuxMountContext="" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241199 5988 reconstruct.go:97] "Volume reconstruction finished" Nov 03 18:38:45.241372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.241207 5988 reconciler.go:29] "Reconciler: start to sync state" Nov 03 18:38:45.262776 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.262728 5988 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 18:38:45.264775 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.264746 5988 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 18:38:45.264775 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.264771 5988 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 18:38:45.264991 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.264799 5988 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 18:38:45.264991 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.264840 5988 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 18:38:45.304329 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.304244 5988 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="ef69bf03eebe06081e33bcb0ed8f20c0ab0ec7c169ef5f31f12fbf60c1a3318c" exitCode=0 Nov 03 18:38:45.304329 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.304285 5988 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="7a742cce491eec5b1b1cad8a77149176846b2e7d645ab77d683612e11d20b874" exitCode=0 Nov 03 18:38:45.304329 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.304296 5988 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="a71cab2f636fda979a8034ae7e6cfc1486ccfa61bd94c8f9d7db1d561baae5bb" exitCode=0 Nov 03 18:38:45.304329 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.304308 5988 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="ba655e881d0b48c7fcfdcff3b6927b83ad93329fc0dd3866a6695d0b19c2b4b4" exitCode=0 Nov 03 18:38:45.304329 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.304317 5988 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="6de743d19d8ffe42aa3df62270b6b06b4b61adf31781e73c29c0249d58190412" exitCode=0 Nov 03 18:38:45.304329 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.304326 5988 generic.go:356] "Generic (PLEG): container finished" podID="0f0cd7d8-5868-443d-88a4-8d2e464631e9" containerID="a00cc96a751139d44617e4956317a8d1b36b4da6268b7a380f340fc88ae3e08e" exitCode=0 Nov 03 18:38:45.319098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.319062 5988 generic.go:356] "Generic (PLEG): container finished" podID="02282d0e3b70f0c274312efbaf7c57b7" containerID="7b5255ee7e685a3b3da48de2e61da2c46207dc8f5c1e26584602cde5003f793f" exitCode=0 Nov 03 18:38:45.325914 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.325881 5988 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" exitCode=0 Nov 03 18:38:45.328598 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.328574 5988 generic.go:356] "Generic (PLEG): container finished" podID="b07cbb8d-1cfb-425b-8de7-e8ae88befa55" containerID="b812a8225a268642ab6dbd772c5c8cc638b136043b91403175371cee0f67aef2" exitCode=0 Nov 03 18:38:45.337647 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.337618 5988 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-161.us-west-2.compute.internal\" not found" Nov 03 18:38:45.349602 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.349577 5988 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0" exitCode=0 Nov 03 18:38:45.364970 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.364943 5988 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 18:38:45.380600 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.380568 5988 manager.go:324] Recovery completed Nov 03 18:38:45.400082 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.400051 5988 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:38:45.400216 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.400139 5988 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 18:38:45.400216 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.400189 5988 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:38:45.401089 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.401068 5988 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:38:45.416259 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416236 5988 cpu_manager.go:226] "Starting" policy="none" Nov 03 18:38:45.416259 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416255 5988 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 18:38:45.416414 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416278 5988 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 18:38:45.416460 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416443 5988 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 18:38:45.416518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416458 5988 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 18:38:45.416518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416483 5988 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 18:38:45.416518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416492 5988 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 18:38:45.416518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416500 5988 policy_none.go:50] "Start" Nov 03 18:38:45.416518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416511 5988 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 18:38:45.416675 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416524 5988 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 18:38:45.416735 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416718 5988 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 18:38:45.416735 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416729 5988 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 18:38:45.416789 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.416738 5988 policy_none.go:44] "Start" Nov 03 18:38:45.423823 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.423267 5988 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 18:38:45.431300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.431275 5988 manager.go:341] "Starting Device Plugin manager" Nov 03 18:38:45.431402 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.431309 5988 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 18:38:45.431402 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.431322 5988 server.go:87] "Starting device plugin registration server" Nov 03 18:38:45.431649 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.431633 5988 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 18:38:45.431715 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.431651 5988 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 18:38:45.431897 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.431879 5988 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 18:38:45.432010 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.431975 5988 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 18:38:45.432010 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.432010 5988 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 18:38:45.432131 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.432015 5988 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 18:38:45.432180 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.432081 5988 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:38:45.432002027Z","UUID":"3b67a51f-dbc8-4e22-aed3-34e27812a3db","Handler":null,"Name":"","Endpoint":""} Nov 03 18:38:45.434057 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.433821 5988 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 18:38:45.434168 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.434137 5988 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 18:38:45.434226 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.434174 5988 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 18:38:45.492896 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.492833 5988 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 18:38:45.532575 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.532523 5988 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 18:38:45.534970 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.534943 5988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 18:38:45.535084 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.534990 5988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 18:38:45.535084 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.535007 5988 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-161.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 18:38:45.535174 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.535094 5988 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.565874 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.565751 5988 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal"] Nov 03 18:38:45.566731 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.566712 5988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.566847 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.566737 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"3a04c408b2b26c5648ca41d83d5e6490","Type":"ContainerStarted","Data":"4a3bcbb778ed34a4aed4ae1a15acf9acdf6c1cfe83798121c3b4459eddfb5c8d"} Nov 03 18:38:45.566847 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.566781 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"3a04c408b2b26c5648ca41d83d5e6490","Type":"ContainerStarted","Data":"7d33161e83cc5b354136faba1ac65d6a94d9e75a1675088632a0aa5eb967863e"} Nov 03 18:38:45.566847 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.566797 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"02282d0e3b70f0c274312efbaf7c57b7","Type":"ContainerStarted","Data":"8cb32b69011f638c75aa999d73e826e2f9f24caa066b4f684bcbaf3c7e7dbca6"} Nov 03 18:38:45.566847 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.566810 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"02282d0e3b70f0c274312efbaf7c57b7","Type":"ContainerDied","Data":"7b5255ee7e685a3b3da48de2e61da2c46207dc8f5c1e26584602cde5003f793f"} Nov 03 18:38:45.566847 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.566828 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" event={"ID":"02282d0e3b70f0c274312efbaf7c57b7","Type":"ContainerStarted","Data":"437c8f6abdbff0a017385cc6168fe6e2f4ef9ec37466210027eeebc398672bec"} Nov 03 18:38:45.567037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.566944 5988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.635256 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.635213 5988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:38:45.635443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.635272 5988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:38:45.635443 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.635326 5988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.635443 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:45.635329 5988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.651015 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.650983 5988 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.651169 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.651073 5988 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.701337 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.701298 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.701518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.701346 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a04c408b2b26c5648ca41d83d5e6490-config\") pod \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"3a04c408b2b26c5648ca41d83d5e6490\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.701518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.701376 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.802441 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.802405 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.802611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.802411 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.802611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.802496 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a04c408b2b26c5648ca41d83d5e6490-config\") pod \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"3a04c408b2b26c5648ca41d83d5e6490\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.802611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.802537 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.802611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.802554 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02282d0e3b70f0c274312efbaf7c57b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"02282d0e3b70f0c274312efbaf7c57b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:45.802611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:45.802591 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a04c408b2b26c5648ca41d83d5e6490-config\") pod \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" (UID: \"3a04c408b2b26c5648ca41d83d5e6490\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:46.200345 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.200297 5988 apiserver.go:52] "Watching apiserver" Nov 03 18:38:46.354182 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.354153 5988 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 18:38:46.355011 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.354982 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-wlzkp","kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv","openshift-monitoring/telemeter-client-74f6bf747d-rsr85","openshift-image-registry/node-ca-npkpz","openshift-multus/multus-8xwzz","openshift-console/downloads-6dd789c8b7-g98j2","openshift-monitoring/alertmanager-main-0","kube-system/global-pull-secret-syncer-2np47","kube-system/konnectivity-agent-5sdxh","openshift-insights/insights-runtime-extractor-p9mbd","openshift-multus/network-metrics-daemon-df7m6","openshift-monitoring/node-exporter-vj6dq","openshift-monitoring/thanos-querier-c8cc96c46-k9l2m","openshift-network-operator/iptables-alerter-blnln","openshift-dns/node-resolver-p2wnl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-7gbhd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs","openshift-monitoring/prometheus-operator-9cd89b74c-txxgc","openshift-ingress-canary/ingress-canary-4fd6d","openshift-dns/dns-default-js86h","openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r","openshift-monitoring/prometheus-k8s-0","openshift-network-diagnostics/network-check-target-gl7nv","openshift-monitoring/metrics-server-7db59cdcd6-pqldv","openshift-cluster-node-tuning-operator/tuned-j9fq8"] Nov 03 18:38:46.368542 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.368499 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:46.369229 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.369179 5988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:46.369376 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.369310 5988 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:46.405753 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.405718 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d4mhq\"" Nov 03 18:38:46.405923 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.405809 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hkfh8\"" Nov 03 18:38:46.405923 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.405810 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 18:38:46.405923 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.405916 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p4grw\"" Nov 03 18:38:46.406751 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.406732 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.406860 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.406763 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.406860 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.406818 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.406940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.406888 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.440115 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.440080 5988 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 18:38:46.454546 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.454473 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 18:38:46.456918 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.456874 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 18:38:46.456918 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.456882 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 18:38:46.457151 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.456974 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.457151 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457006 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.457258 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457212 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.457258 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457243 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.457434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457278 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.457434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457393 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 18:38:46.457434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457403 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.457538 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457439 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 18:38:46.457810 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457792 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 18:38:46.457903 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457815 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.457903 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.457839 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.458260 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458241 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fft52\"" Nov 03 18:38:46.458260 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458256 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 18:38:46.458403 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458301 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.458403 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458306 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.458517 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458496 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zsb8x\"" Nov 03 18:38:46.458618 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458607 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 18:38:46.458738 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458721 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cxqpk\"" Nov 03 18:38:46.458850 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458825 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 18:38:46.458919 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458886 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-btttq\"" Nov 03 18:38:46.458997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.458975 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cxjph\"" Nov 03 18:38:46.462997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.462959 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 18:38:46.463118 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.463100 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 18:38:46.465909 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.465890 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 18:38:46.466036 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.465935 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 18:38:46.466086 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466046 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 18:38:46.466298 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466237 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.466298 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466263 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 18:38:46.466410 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466308 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 18:38:46.466410 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466316 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.466410 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466332 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.466534 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466425 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.466573 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.466560 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 18:38:46.467375 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.467360 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 18:38:46.469941 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.469924 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.470818 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.470799 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 18:38:46.471026 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.470985 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 18:38:46.504559 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.504526 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 18:38:46.505257 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.505227 5988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:38:46.505423 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:46.505300 5988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:46.505876 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.505848 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-flzhh\"" Nov 03 18:38:46.506156 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.506140 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 18:38:46.506452 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.506438 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 18:38:46.506704 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.506669 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gfh9q\"" Nov 03 18:38:46.507370 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.507352 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 18:38:46.507781 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.507765 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 18:38:46.508019 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.508002 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 18:38:46.508572 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.505848 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 18:38:46.509306 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.509290 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 18:38:46.509364 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.509353 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r6plj\"" Nov 03 18:38:46.509512 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.509501 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 18:38:46.509546 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.509537 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-jq1vc25cvje4\"" Nov 03 18:38:46.509714 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.509701 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3c2nv2isc8l0t\"" Nov 03 18:38:46.510117 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.510098 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 18:38:46.510153 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.510140 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 18:38:46.510710 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.510669 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 18:38:46.510799 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.510708 5988 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 18:38:46.510799 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:46.510786 5988 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal" Nov 03 18:38:46.511013 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.510997 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 18:38:46.511096 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.511085 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 18:38:46.511221 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.511210 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczj\"" Nov 03 18:38:46.511461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.511449 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 18:38:46.511961 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.511942 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r88kn\"" Nov 03 18:38:46.512088 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.512070 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 18:38:46.512088 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.512083 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 18:38:46.512281 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.512264 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 18:38:46.512328 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.512293 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 18:38:46.512608 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.512553 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 18:38:46.513492 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513473 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-grpc-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.513579 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513506 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snlf\" (UniqueName: \"kubernetes.io/projected/c026965c-43dc-41e4-a73a-90ba867305a5-kube-api-access-6snlf\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.513579 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513532 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.513579 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513572 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.513767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513599 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.513767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513631 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.513767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513658 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.513767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513707 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-sys-fs\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.513767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513735 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3c563d-6155-43a3-bff6-5e95b9cee38d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.513767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513762 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513787 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513812 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-crio-socket\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513838 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fjt\" (UniqueName: \"kubernetes.io/projected/0f0cd7d8-5868-443d-88a4-8d2e464631e9-kube-api-access-h9fjt\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513863 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513887 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513899 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513912 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lzn26\"" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513915 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c026965c-43dc-41e4-a73a-90ba867305a5-metrics-client-ca\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.514053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514028 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-etc-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514066 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42604250-348d-4471-8d8a-d377e8986eaf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514096 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-registration-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514122 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-wtmp\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514145 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-kubelet\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514171 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736cce59-1d70-4b20-975d-58d36563698f-host-slash\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514198 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514224 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-slash\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514250 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtjx\" (UniqueName: \"kubernetes.io/projected/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-kube-api-access-mhtjx\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514275 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-device-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514274 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-grpc-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514294 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f3c563d-6155-43a3-bff6-5e95b9cee38d-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513987 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513790 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514074 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 18:38:46.514473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514453 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514299 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfq9n\" (UniqueName: \"kubernetes.io/projected/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kube-api-access-sfq9n\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514490 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514533 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rdgcx\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.513865 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dvl9\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514581 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514593 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514418 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514727 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514732 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b7l8x\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514746 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514799 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514822 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514829 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514828 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514857 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514864 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514877 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-var-lib-kubelet\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514904 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jk4jl\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514906 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cni-binary-copy\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514923 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qc7sj\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514937 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514980 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515002 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d6l69\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515012 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515019 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515047 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515074 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515083 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-88hbk\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515119 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zjfk4\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515138 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515162 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515170 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515176 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2n24p\"" Nov 03 18:38:46.515176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515198 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vgjjr\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514935 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-metrics-server-audit-profiles\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.514910 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515220 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515276 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-var-lib-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515300 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515308 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515308 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-metrics-client-ca\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515346 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515469 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515490 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cni-binary-copy\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515502 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cnibin\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515514 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515477 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515525 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515535 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515506 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515581 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nx4fl\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515761 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515823 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-metrics-client-ca\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515845 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-metrics-client-ca\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515854 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515930 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515975 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.515988 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9c3kkmaf6dql\"" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516090 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-metrics-client-ca\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516119 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-accelerators-collector-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516175 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516214 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516214 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516224 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516304 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516327 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxcg\" (UniqueName: \"kubernetes.io/projected/74f5b2ca-aa33-444a-bd6f-56df2278d099-kube-api-access-jgxcg\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516379 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516441 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-accelerators-collector-config\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516515 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516558 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516570 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-textfile\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516622 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-textfile\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516626 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14054e49-309c-46d4-ba51-5a2ad669f1a0-host\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:38:46.516642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516659 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7601adc3-f067-48c5-b556-58b607361367-tmp\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516767 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516793 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-kubelet\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516822 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516900 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7601adc3-f067-48c5-b556-58b607361367-tmp\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516926 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516944 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516960 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-kubernetes\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.516988 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmjb\" (UniqueName: \"kubernetes.io/projected/38dd707d-5338-4e99-a3b4-49380731aebb-kube-api-access-szmjb\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517007 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736cce59-1d70-4b20-975d-58d36563698f-iptables-alerter-script\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517044 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06841536-c400-4c0f-8d46-5d8e24ba9c6d-konnectivity-ca\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517065 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517162 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-netd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517211 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/06841536-c400-4c0f-8d46-5d8e24ba9c6d-konnectivity-ca\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517248 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517275 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/736cce59-1d70-4b20-975d-58d36563698f-iptables-alerter-script\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517295 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smrrx\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517332 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-kubelet-config\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517332 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517392 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517432 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cnibin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517468 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08020da4-3f4c-4eef-8e5f-126abd177058-tmp-dir\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517502 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517525 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517561 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysconfig\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517607 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08020da4-3f4c-4eef-8e5f-126abd177058-tmp-dir\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517613 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-socket-dir-parent\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517672 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-tls\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517735 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-federate-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517831 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-socket-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517849 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517869 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-tls\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517893 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517916 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.517934 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-tls\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518074 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518151 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-tls\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518170 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-federate-client-tls\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518223 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl46\" (UniqueName: \"kubernetes.io/projected/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-api-access-gwl46\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518236 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518301 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518312 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-systemd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.518380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518349 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518438 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-ovn\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518485 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518553 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-script-lib\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518624 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518652 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518677 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518676 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518731 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74f5b2ca-aa33-444a-bd6f-56df2278d099-audit-log\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.518762 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rfm\" (UniqueName: \"kubernetes.io/projected/4322a265-eccb-4e0b-89e3-8844e35f3b6e-kube-api-access-p5rfm\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519054 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519109 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519123 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519153 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519198 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519220 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74f5b2ca-aa33-444a-bd6f-56df2278d099-audit-log\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519252 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxgs4\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519277 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519292 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519336 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519358 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519374 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519420 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519456 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519470 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-script-lib\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519490 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519509 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519522 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08020da4-3f4c-4eef-8e5f-126abd177058-metrics-tls\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519560 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf78j\" (UniqueName: \"kubernetes.io/projected/42604250-348d-4471-8d8a-d377e8986eaf-kube-api-access-pf78j\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519598 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-netns\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519604 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519746 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-etc-kubernetes\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519813 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08020da4-3f4c-4eef-8e5f-126abd177058-config-volume\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519850 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzv4\" (UniqueName: \"kubernetes.io/projected/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-kube-api-access-fvzv4\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519881 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519898 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519905 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.519974 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4322a265-eccb-4e0b-89e3-8844e35f3b6e-metrics-certs\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520037 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520053 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/08020da4-3f4c-4eef-8e5f-126abd177058-config-volume\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520078 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-client-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520110 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-config\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520138 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520157 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520158 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-env-overrides\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520200 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520199 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/08020da4-3f4c-4eef-8e5f-126abd177058-metrics-tls\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520267 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520298 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovnkube-config\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520310 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-lib-modules\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520328 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-env-overrides\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520359 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxsq\" (UniqueName: \"kubernetes.io/projected/7601adc3-f067-48c5-b556-58b607361367-kube-api-access-2nxsq\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:46.520339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520379 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-client-ca-bundle\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520404 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktm4\" (UniqueName: \"kubernetes.io/projected/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-kube-api-access-5ktm4\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520431 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520456 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f140b25-ea89-4670-8e35-047a4bd190ac-cert\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520453 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520494 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-daemon-config\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520520 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520545 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520572 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxjv\" (UniqueName: \"kubernetes.io/projected/1f140b25-ea89-4670-8e35-047a4bd190ac-kube-api-access-qfxjv\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520594 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520617 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-sys\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520638 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520643 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltb7\" (UniqueName: \"kubernetes.io/projected/14054e49-309c-46d4-ba51-5a2ad669f1a0-kube-api-access-vltb7\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520669 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-k8s-cni-cncf-io\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520715 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-multus-certs\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520745 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520763 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-daemon-config\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520767 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f140b25-ea89-4670-8e35-047a4bd190ac-cert\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520775 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520778 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520809 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-system-cni-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520812 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.520999 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-tls\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521001 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmnm\" (UniqueName: \"kubernetes.io/projected/736cce59-1d70-4b20-975d-58d36563698f-kube-api-access-2kmnm\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521036 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521036 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521073 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521112 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-node-log\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521202 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521260 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovn-node-metrics-cert\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521308 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521358 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521386 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-host\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521413 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwdr\" (UniqueName: \"kubernetes.io/projected/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-kube-api-access-wcwdr\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521440 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-bin\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521467 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-serving-certs-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521521 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521550 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-ovn-node-metrics-cert\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521557 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-sys\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521590 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-root\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521625 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521630 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521648 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-serving-certs-ca-bundle\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521705 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521741 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521745 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-conf\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521797 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-etc-tuned\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521828 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-bin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521842 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521859 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-hostroot\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521889 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlr8v\" (UniqueName: \"kubernetes.io/projected/9b32f9a9-d324-4501-9c23-4e93b00863aa-kube-api-access-hlr8v\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521907 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-data-volume\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521924 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-dbus\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521889 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-etc-tuned\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521946 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.521972 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522001 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-data-volume\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522009 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-conf-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522121 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvqf\" (UniqueName: \"kubernetes.io/projected/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-api-access-psvqf\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522193 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-client-certs\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522230 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-hosts-file\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522317 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522403 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522433 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-tmp\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522432 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d25cdfb-758e-47fa-adbc-8f6953a7be99-original-pull-secret\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522475 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx8d\" (UniqueName: \"kubernetes.io/projected/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-kube-api-access-bfx8d\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522519 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38dd707d-5338-4e99-a3b4-49380731aebb-tmp\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522516 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74f5b2ca-aa33-444a-bd6f-56df2278d099-secret-metrics-server-client-certs\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522533 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mqm\" (UniqueName: \"kubernetes.io/projected/08020da4-3f4c-4eef-8e5f-126abd177058-kube-api-access-b4mqm\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522620 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-netns\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.522660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522722 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-log-socket\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522781 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522903 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522915 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.522955 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-multus\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523026 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523033 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523073 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-tmp-dir\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523141 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523196 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-tmp-dir\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523256 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523327 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523356 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523383 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-modprobe-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523406 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523434 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06841536-c400-4c0f-8d46-5d8e24ba9c6d-agent-certs\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523460 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-systemd-units\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523485 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14054e49-309c-46d4-ba51-5a2ad669f1a0-serviceca\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523500 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523516 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523740 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b32f9a9-d324-4501-9c23-4e93b00863aa-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523789 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.523902 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f0cd7d8-5868-443d-88a4-8d2e464631e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524040 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/06841536-c400-4c0f-8d46-5d8e24ba9c6d-agent-certs\") pod \"konnectivity-agent-5sdxh\" (UID: \"06841536-c400-4c0f-8d46-5d8e24ba9c6d\") " pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524211 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c026965c-43dc-41e4-a73a-90ba867305a5-metrics-client-ca\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524252 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524279 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c026965c-43dc-41e4-a73a-90ba867305a5-tmp\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524307 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-systemd\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524336 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-run\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524361 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-system-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524386 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b32f9a9-d324-4501-9c23-4e93b00863aa-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524496 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14054e49-309c-46d4-ba51-5a2ad669f1a0-serviceca\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524663 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b32f9a9-d324-4501-9c23-4e93b00863aa-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524733 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c026965c-43dc-41e4-a73a-90ba867305a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524781 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c026965c-43dc-41e4-a73a-90ba867305a5-tmp\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524861 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524956 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.524953 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42604250-348d-4471-8d8a-d377e8986eaf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.525076 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42604250-348d-4471-8d8a-d377e8986eaf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.525444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.525222 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74f5b2ca-aa33-444a-bd6f-56df2278d099-metrics-server-audit-profiles\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.528346 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.528321 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.528522 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.528506 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.529962 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.529942 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.573900 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.573865 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxcg\" (UniqueName: \"kubernetes.io/projected/74f5b2ca-aa33-444a-bd6f-56df2278d099-kube-api-access-jgxcg\") pod \"metrics-server-7db59cdcd6-pqldv\" (UID: \"74f5b2ca-aa33-444a-bd6f-56df2278d099\") " pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:38:46.575494 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.575469 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfq9n\" (UniqueName: \"kubernetes.io/projected/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kube-api-access-sfq9n\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.576476 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.576448 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmjb\" (UniqueName: \"kubernetes.io/projected/38dd707d-5338-4e99-a3b4-49380731aebb-kube-api-access-szmjb\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.576709 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.576668 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxgs4\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\") pod \"alertmanager-main-0\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:38:46.579275 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.579251 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxsq\" (UniqueName: \"kubernetes.io/projected/7601adc3-f067-48c5-b556-58b607361367-kube-api-access-2nxsq\") pod \"downloads-6dd789c8b7-g98j2\" (UID: \"7601adc3-f067-48c5-b556-58b607361367\") " pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:38:46.579956 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.579934 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl46\" (UniqueName: \"kubernetes.io/projected/1f3c563d-6155-43a3-bff6-5e95b9cee38d-kube-api-access-gwl46\") pod \"kube-state-metrics-5dc8d7d89b-5ch4r\" (UID: \"1f3c563d-6155-43a3-bff6-5e95b9cee38d\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5ch4r" Nov 03 18:38:46.580064 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.579998 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fjt\" (UniqueName: \"kubernetes.io/projected/0f0cd7d8-5868-443d-88a4-8d2e464631e9-kube-api-access-h9fjt\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.581342 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.581319 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rfm\" (UniqueName: \"kubernetes.io/projected/4322a265-eccb-4e0b-89e3-8844e35f3b6e-kube-api-access-p5rfm\") pod \"network-metrics-daemon-df7m6\" (UID: \"4322a265-eccb-4e0b-89e3-8844e35f3b6e\") " pod="openshift-multus/network-metrics-daemon-df7m6" Nov 03 18:38:46.581517 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.581491 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtjx\" (UniqueName: \"kubernetes.io/projected/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-kube-api-access-mhtjx\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:38:46.582121 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.582096 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smrrx\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\") pod \"prometheus-k8s-0\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:46.584875 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.584850 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf78j\" (UniqueName: \"kubernetes.io/projected/42604250-348d-4471-8d8a-d377e8986eaf-kube-api-access-pf78j\") pod \"openshift-state-metrics-7bd7489957-wj6cv\" (UID: \"42604250-348d-4471-8d8a-d377e8986eaf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wj6cv" Nov 03 18:38:46.585416 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.585395 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snlf\" (UniqueName: \"kubernetes.io/projected/c026965c-43dc-41e4-a73a-90ba867305a5-kube-api-access-6snlf\") pod \"thanos-querier-c8cc96c46-k9l2m\" (UID: \"c026965c-43dc-41e4-a73a-90ba867305a5\") " pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:46.585475 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.585400 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzv4\" (UniqueName: \"kubernetes.io/projected/3d64f9eb-a543-4518-8d1d-0a27cac5a88a-kube-api-access-fvzv4\") pod \"telemeter-client-74f6bf747d-rsr85\" (UID: \"3d64f9eb-a543-4518-8d1d-0a27cac5a88a\") " pod="openshift-monitoring/telemeter-client-74f6bf747d-rsr85" Nov 03 18:38:46.590447 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.590414 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:46.606805 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.606770 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktm4\" (UniqueName: \"kubernetes.io/projected/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-kube-api-access-5ktm4\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.612286 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.612250 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mqm\" (UniqueName: \"kubernetes.io/projected/08020da4-3f4c-4eef-8e5f-126abd177058-kube-api-access-b4mqm\") pod \"dns-default-js86h\" (UID: \"08020da4-3f4c-4eef-8e5f-126abd177058\") " pod="openshift-dns/dns-default-js86h" Nov 03 18:38:46.619129 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.619107 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwdr\" (UniqueName: \"kubernetes.io/projected/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-kube-api-access-wcwdr\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.619359 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.619318 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmnm\" (UniqueName: \"kubernetes.io/projected/736cce59-1d70-4b20-975d-58d36563698f-kube-api-access-2kmnm\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:38:46.619412 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.619355 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zw8\" (UniqueName: \"kubernetes.io/projected/aedaa821-137b-4371-9a23-c4a2890cd484-kube-api-access-c5zw8\") pod \"network-check-target-gl7nv\" (UID: \"aedaa821-137b-4371-9a23-c4a2890cd484\") " pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:46.619512 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.619495 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxjv\" (UniqueName: \"kubernetes.io/projected/1f140b25-ea89-4670-8e35-047a4bd190ac-kube-api-access-qfxjv\") pod \"ingress-canary-4fd6d\" (UID: \"1f140b25-ea89-4670-8e35-047a4bd190ac\") " pod="openshift-ingress-canary/ingress-canary-4fd6d" Nov 03 18:38:46.623578 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.623557 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltb7\" (UniqueName: \"kubernetes.io/projected/14054e49-309c-46d4-ba51-5a2ad669f1a0-kube-api-access-vltb7\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:38:46.625614 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625591 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-node-log\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.625710 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625629 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.625710 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625656 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-host\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.625710 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625703 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-bin\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625725 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625722 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-node-log\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625732 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-sys\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625768 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-host\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625777 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-sys\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625800 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-bin\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625824 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-root\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.625851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625851 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-conf\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625875 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-bin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625900 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-hostroot\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625905 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-root\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625921 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-dbus\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625945 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-conf-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625951 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-bin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625969 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-hosts-file\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625983 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-hostroot\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625996 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-netns\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.625999 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-conf\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626021 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-log-socket\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626040 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c91d4f5-0557-4eda-abb2-9f49e237d4b5-hosts-file\") pod \"node-resolver-p2wnl\" (UID: \"2c91d4f5-0557-4eda-abb2-9f49e237d4b5\") " pod="openshift-dns/node-resolver-p2wnl" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626044 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-netns\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626062 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-conf-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626075 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-log-socket\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626030 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-dbus\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626120 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-multus\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626148 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-modprobe-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626151 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-cni-multus\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626170 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626198 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-systemd-units\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626238 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-systemd\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626259 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-run\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626265 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-modprobe-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626281 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-systemd-units\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626292 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-system-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626298 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysctl-d\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626308 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-run\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626311 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-systemd\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626312 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626320 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-system-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626345 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-etc-selinux\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626348 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-sys-fs\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626380 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-crio-socket\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626398 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-sys-fs\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626410 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-etc-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626427 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-registration-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626430 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-crio-socket\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626442 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-wtmp\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626470 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-kubelet\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626477 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b07cbb8d-1cfb-425b-8de7-e8ae88befa55-node-exporter-wtmp\") pod \"node-exporter-vj6dq\" (UID: \"b07cbb8d-1cfb-425b-8de7-e8ae88befa55\") " pod="openshift-monitoring/node-exporter-vj6dq" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626514 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736cce59-1d70-4b20-975d-58d36563698f-host-slash\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626524 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-etc-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626554 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/736cce59-1d70-4b20-975d-58d36563698f-host-slash\") pod \"iptables-alerter-blnln\" (UID: \"736cce59-1d70-4b20-975d-58d36563698f\") " pod="openshift-network-operator/iptables-alerter-blnln" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626564 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-registration-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626555 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-var-lib-kubelet\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626571 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-slash\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626597 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-slash\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626601 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-device-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626636 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-device-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626662 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-var-lib-kubelet\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626710 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-var-lib-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626730 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626747 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cnibin\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626766 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626792 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-var-lib-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626792 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14054e49-309c-46d4-ba51-5a2ad669f1a0-host\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626819 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14054e49-309c-46d4-ba51-5a2ad669f1a0-host\") pod \"node-ca-npkpz\" (UID: \"14054e49-309c-46d4-ba51-5a2ad669f1a0\") " pod="openshift-image-registry/node-ca-npkpz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626823 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-kubelet\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626856 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-kubernetes\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626864 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626874 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-netd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626893 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-kubelet-config\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626907 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cnibin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626923 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysconfig\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626926 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-cni-dir\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626942 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-socket-dir-parent\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626959 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-socket-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626963 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-cni-netd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.626936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626975 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-systemd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626995 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-ovn\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627007 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-var-lib-kubelet\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627016 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627041 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627045 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-kubelet\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627055 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-netns\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627070 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-etc-kubernetes\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627076 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-kubernetes\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627098 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-lib-modules\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627109 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7933f6c-3101-4ee6-af60-ecec7ac89f03-socket-dir\") pod \"aws-ebs-csi-driver-node-qhhcs\" (UID: \"a7933f6c-3101-4ee6-af60-ecec7ac89f03\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qhhcs" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627124 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-sys\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627140 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-k8s-cni-cncf-io\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627154 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-multus-certs\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627170 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-system-cni-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627175 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-cnibin\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627192 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627211 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-etc-sysconfig\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627251 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-openvswitch\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627276 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-systemd\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627291 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-sys\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627306 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-run-ovn\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627315 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-k8s-cni-cncf-io\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627140 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d25cdfb-758e-47fa-adbc-8f6953a7be99-kubelet-config\") pod \"global-pull-secret-syncer-2np47\" (UID: \"4d25cdfb-758e-47fa-adbc-8f6953a7be99\") " pod="kube-system/global-pull-secret-syncer-2np47" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627331 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46f7e70a-abc0-49c1-a7a5-15ef4ff91925-host-run-ovn-kubernetes\") pod \"ovnkube-node-7gbhd\" (UID: \"46f7e70a-abc0-49c1-a7a5-15ef4ff91925\") " pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627351 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-multus-certs\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.626894 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-cnibin\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627379 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627399 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f0cd7d8-5868-443d-88a4-8d2e464631e9-system-cni-dir\") pod \"multus-additional-cni-plugins-wlzkp\" (UID: \"0f0cd7d8-5868-443d-88a4-8d2e464631e9\") " pod="openshift-multus/multus-additional-cni-plugins-wlzkp" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627406 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-host-run-netns\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627251 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-multus-socket-dir-parent\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627435 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-etc-kubernetes\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.628523 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.627490 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38dd707d-5338-4e99-a3b4-49380731aebb-lib-modules\") pod \"tuned-j9fq8\" (UID: \"38dd707d-5338-4e99-a3b4-49380731aebb\") " pod="openshift-cluster-node-tuning-operator/tuned-j9fq8" Nov 03 18:38:46.637811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.637785 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx8d\" (UniqueName: \"kubernetes.io/projected/8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d-kube-api-access-bfx8d\") pod \"multus-8xwzz\" (UID: \"8dca58b4-bcb5-4e7e-8eaa-d42b381ef76d\") " pod="openshift-multus/multus-8xwzz" Nov 03 18:38:46.638995 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.638970 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvqf\" (UniqueName: \"kubernetes.io/projected/43875fcc-00e2-40f4-b610-f4c8aa6ac7ce-kube-api-access-psvqf\") pod \"insights-runtime-extractor-p9mbd\" (UID: \"43875fcc-00e2-40f4-b610-f4c8aa6ac7ce\") " pod="openshift-insights/insights-runtime-extractor-p9mbd" Nov 03 18:38:46.639296 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.639278 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlr8v\" (UniqueName: \"kubernetes.io/projected/9b32f9a9-d324-4501-9c23-4e93b00863aa-kube-api-access-hlr8v\") pod \"prometheus-operator-9cd89b74c-txxgc\" (UID: \"9b32f9a9-d324-4501-9c23-4e93b00863aa\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-txxgc" Nov 03 18:38:46.673943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:46.673921 5988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:38:47.680543 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:47.680504 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-js86h" Nov 03 18:38:47.806436 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:47.806398 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:38:48.510984 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:48.510944 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:48.543649 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:48.543615 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c8cc96c46-k9l2m" Nov 03 18:38:49.712732 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:49.712697 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:49.917134 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:49.917105 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:50.100185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:50.100147 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:50.413013 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:50.412934 5988 log.go:32] "PullImage from image service failed" err="rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36c3da5821b9e51334e9c523aca7fca6a9ef8b853fd4f238b908b086b129619d" Nov 03 18:38:50.413199 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:50.413175 5988 kuberuntime_manager.go:1707] "Unhandled Error" err="container download-server start failed in pod downloads-6dd789c8b7-g98j2_openshift-console(7601adc3-f067-48c5-b556-58b607361367): ErrImagePull: rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" logger="UnhandledError" Nov 03 18:38:50.413311 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:50.413216 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"download-server\" with ErrImagePull: \"rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled\"" pod="openshift-console/downloads-6dd789c8b7-g98j2" podUID="7601adc3-f067-48c5-b556-58b607361367" Nov 03 18:38:51.407927 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:51.407894 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:38:54.586553 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:54.586478 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5sdxh" Nov 03 18:38:54.725477 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:54.725454 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:54.725661 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:54.725650 5988 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 18:38:55.033130 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:55.033062 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7gbhd" Nov 03 18:38:55.935376 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:38:55.935343 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:38:58.729199 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:38:58.729170 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7db59cdcd6-pqldv" Nov 03 18:39:00.761568 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:00.761532 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:00.982582 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:00.982542 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:01.732022 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:01.731978 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:05.445814 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:05.445728 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-g98j2" event={"ID":"7601adc3-f067-48c5-b556-58b607361367","Type":"ContainerStarted","Data":"4f3e903d68e1dfad0f315bb31be5b7f4554093ebb14590255d4eae3ff73657b2"} Nov 03 18:39:05.446284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:05.446031 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:39:05.457044 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:05.457017 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-g98j2" Nov 03 18:39:06.116210 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:06.116176 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:15.779977 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:15.779941 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:16.018900 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:16.018872 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:17.809095 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:17.809068 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gl7nv" Nov 03 18:39:22.713439 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:22.713398 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:25.934288 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:39:25.934249 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:39:40.401853 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:40.401811 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:50.119172 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:50.119143 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:39:50.133853 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:50.133830 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:39:54.499121 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:54.499085 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:54.705590 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:54.705553 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:55.935639 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:39:55.935602 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:39:57.722294 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:57.722257 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:58.185547 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:58.185519 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:58.397240 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:39:58.397200 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:02.207130 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.207094 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:40:02.207997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.207953 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="alertmanager" containerID="cri-o://0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336" gracePeriod=120 Nov 03 18:40:02.208157 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.207978 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-metric" containerID="cri-o://e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261" gracePeriod=120 Nov 03 18:40:02.208234 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.208010 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy" containerID="cri-o://a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999" gracePeriod=120 Nov 03 18:40:02.208281 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.208014 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="config-reloader" containerID="cri-o://3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446" gracePeriod=120 Nov 03 18:40:02.208322 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.207987 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-web" containerID="cri-o://82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7" gracePeriod=120 Nov 03 18:40:02.208364 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.208032 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="prom-label-proxy" containerID="cri-o://28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840" gracePeriod=120 Nov 03 18:40:02.661448 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661415 5988 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840" exitCode=0 Nov 03 18:40:02.661448 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661439 5988 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999" exitCode=0 Nov 03 18:40:02.661448 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661446 5988 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446" exitCode=0 Nov 03 18:40:02.661448 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661454 5988 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336" exitCode=0 Nov 03 18:40:02.661719 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661491 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840"} Nov 03 18:40:02.661719 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661552 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999"} Nov 03 18:40:02.661719 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661579 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446"} Nov 03 18:40:02.661719 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:02.661604 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336"} Nov 03 18:40:03.462755 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.462729 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.564830 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.564775 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.564830 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.564814 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565007 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.564844 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565007 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.564878 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565007 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.564907 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565173 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565008 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565173 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565034 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565173 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565071 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565173 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565111 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565173 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565145 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565474 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565120 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:03.565474 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565193 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565474 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565236 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565474 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565268 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\") pod \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\" (UID: \"ce4e2269-545e-4c5d-b5f9-3e337a0fa843\") " Nov 03 18:40:03.565474 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565337 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:03.565782 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565554 5988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-main-db\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.565782 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.565575 5988 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.566640 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.566575 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:03.568455 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.568268 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "kube-api-access-bxgs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:40:03.570735 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.568600 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:03.571342 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.571100 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:03.571443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.571389 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:03.573937 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.573732 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:40:03.573937 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.573776 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:03.573937 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.573780 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:03.575153 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.575128 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:03.578162 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.578130 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:03.583610 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.583590 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config" pod "ce4e2269-545e-4c5d-b5f9-3e337a0fa843" (UID: "ce4e2269-545e-4c5d-b5f9-3e337a0fa843"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:03.665904 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665877 5988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-tls-assets\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.665904 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665903 5988 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665912 5988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-config-out\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665926 5988 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665963 5988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-web-config\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665972 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-metrics-client-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665982 5988 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-main-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.665991 5988 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-cluster-tls-config\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.666000 5988 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.666009 5988 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.666037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.666018 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxgs4\" (UniqueName: \"kubernetes.io/projected/ce4e2269-545e-4c5d-b5f9-3e337a0fa843-kube-api-access-bxgs4\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:03.668594 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.668571 5988 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261" exitCode=0 Nov 03 18:40:03.668594 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.668593 5988 generic.go:356] "Generic (PLEG): container finished" podID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerID="82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7" exitCode=0 Nov 03 18:40:03.668757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.668627 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261"} Nov 03 18:40:03.668757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.668705 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.668757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.668718 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7"} Nov 03 18:40:03.668757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.668746 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ce4e2269-545e-4c5d-b5f9-3e337a0fa843","Type":"ContainerDied","Data":"6001ab64e54c562f935926276ab7e2a20fabccaed1940c72abdb8ea274e10fe0"} Nov 03 18:40:03.668957 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.668774 5988 scope.go:122] "RemoveContainer" containerID="28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840" Nov 03 18:40:03.676985 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.676968 5988 scope.go:122] "RemoveContainer" containerID="e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261" Nov 03 18:40:03.684395 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.684379 5988 scope.go:122] "RemoveContainer" containerID="a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999" Nov 03 18:40:03.690941 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.690924 5988 scope.go:122] "RemoveContainer" containerID="82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7" Nov 03 18:40:03.697233 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.697217 5988 scope.go:122] "RemoveContainer" containerID="3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446" Nov 03 18:40:03.704282 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.704263 5988 scope.go:122] "RemoveContainer" containerID="0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336" Nov 03 18:40:03.710924 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.710904 5988 scope.go:122] "RemoveContainer" containerID="7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0" Nov 03 18:40:03.717331 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.717317 5988 scope.go:122] "RemoveContainer" containerID="28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840" Nov 03 18:40:03.717584 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:03.717567 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840\": container with ID starting with 28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840 not found: ID does not exist" containerID="28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840" Nov 03 18:40:03.717646 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.717591 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840"} err="failed to get container status \"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840\": rpc error: code = NotFound desc = could not find container \"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840\": container with ID starting with 28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840 not found: ID does not exist" Nov 03 18:40:03.717646 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.717625 5988 scope.go:122] "RemoveContainer" containerID="e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261" Nov 03 18:40:03.717856 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:03.717842 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261\": container with ID starting with e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261 not found: ID does not exist" containerID="e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261" Nov 03 18:40:03.717898 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.717861 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261"} err="failed to get container status \"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261\": rpc error: code = NotFound desc = could not find container \"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261\": container with ID starting with e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261 not found: ID does not exist" Nov 03 18:40:03.717898 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.717874 5988 scope.go:122] "RemoveContainer" containerID="a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999" Nov 03 18:40:03.718063 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:03.718048 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999\": container with ID starting with a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999 not found: ID does not exist" containerID="a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999" Nov 03 18:40:03.718098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718066 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999"} err="failed to get container status \"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999\": rpc error: code = NotFound desc = could not find container \"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999\": container with ID starting with a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999 not found: ID does not exist" Nov 03 18:40:03.718098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718079 5988 scope.go:122] "RemoveContainer" containerID="82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7" Nov 03 18:40:03.718293 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:03.718272 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7\": container with ID starting with 82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7 not found: ID does not exist" containerID="82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7" Nov 03 18:40:03.718334 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718300 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7"} err="failed to get container status \"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7\": rpc error: code = NotFound desc = could not find container \"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7\": container with ID starting with 82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7 not found: ID does not exist" Nov 03 18:40:03.718334 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718317 5988 scope.go:122] "RemoveContainer" containerID="3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446" Nov 03 18:40:03.718535 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:03.718519 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446\": container with ID starting with 3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446 not found: ID does not exist" containerID="3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446" Nov 03 18:40:03.718577 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718537 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446"} err="failed to get container status \"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446\": rpc error: code = NotFound desc = could not find container \"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446\": container with ID starting with 3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446 not found: ID does not exist" Nov 03 18:40:03.718577 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718558 5988 scope.go:122] "RemoveContainer" containerID="0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336" Nov 03 18:40:03.718858 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:03.718844 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336\": container with ID starting with 0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336 not found: ID does not exist" containerID="0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336" Nov 03 18:40:03.718914 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718860 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336"} err="failed to get container status \"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336\": rpc error: code = NotFound desc = could not find container \"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336\": container with ID starting with 0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336 not found: ID does not exist" Nov 03 18:40:03.718914 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.718873 5988 scope.go:122] "RemoveContainer" containerID="7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0" Nov 03 18:40:03.719080 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:03.719062 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0\": container with ID starting with 7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0 not found: ID does not exist" containerID="7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0" Nov 03 18:40:03.719121 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719086 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0"} err="failed to get container status \"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0\": rpc error: code = NotFound desc = could not find container \"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0\": container with ID starting with 7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0 not found: ID does not exist" Nov 03 18:40:03.719121 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719100 5988 scope.go:122] "RemoveContainer" containerID="28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840" Nov 03 18:40:03.719346 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719324 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840"} err="failed to get container status \"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840\": rpc error: code = NotFound desc = could not find container \"28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840\": container with ID starting with 28e0a4693000999c97b799ac2b48c97eb524b5de70adaa774ca9df07bb8d8840 not found: ID does not exist" Nov 03 18:40:03.719346 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719344 5988 scope.go:122] "RemoveContainer" containerID="e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261" Nov 03 18:40:03.719558 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719541 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261"} err="failed to get container status \"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261\": rpc error: code = NotFound desc = could not find container \"e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261\": container with ID starting with e27d1f6e3a0cc109ad7e7f3f843244f1cead8d47d1f27e3ae2d2ed791455c261 not found: ID does not exist" Nov 03 18:40:03.719599 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719559 5988 scope.go:122] "RemoveContainer" containerID="a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999" Nov 03 18:40:03.719798 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719782 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999"} err="failed to get container status \"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999\": rpc error: code = NotFound desc = could not find container \"a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999\": container with ID starting with a14cd50ed684e527b843016a092ce6f951d9f4e0ad29874eec61813addade999 not found: ID does not exist" Nov 03 18:40:03.719798 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719796 5988 scope.go:122] "RemoveContainer" containerID="82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7" Nov 03 18:40:03.719997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719979 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7"} err="failed to get container status \"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7\": rpc error: code = NotFound desc = could not find container \"82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7\": container with ID starting with 82181a115f02005ae16ffa3b6b38ad29e64131b43b30b36dd22e84e77673a0d7 not found: ID does not exist" Nov 03 18:40:03.720034 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.719999 5988 scope.go:122] "RemoveContainer" containerID="3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446" Nov 03 18:40:03.720227 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.720209 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446"} err="failed to get container status \"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446\": rpc error: code = NotFound desc = could not find container \"3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446\": container with ID starting with 3333bb3223da4a0dfb4b42f5e77c10a04a96610f847098d621fff2eec7602446 not found: ID does not exist" Nov 03 18:40:03.720268 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.720228 5988 scope.go:122] "RemoveContainer" containerID="0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336" Nov 03 18:40:03.720457 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.720438 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336"} err="failed to get container status \"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336\": rpc error: code = NotFound desc = could not find container \"0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336\": container with ID starting with 0a6c72d0ea9939dcd90bca7d039bcc054796a4c08ae56cde6e0d891981f62336 not found: ID does not exist" Nov 03 18:40:03.720457 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.720455 5988 scope.go:122] "RemoveContainer" containerID="7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0" Nov 03 18:40:03.720707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.720664 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0"} err="failed to get container status \"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0\": rpc error: code = NotFound desc = could not find container \"7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0\": container with ID starting with 7f540549265d1125fa416d6a208669e00d5b851a878bf7ca34debecf2c4f03a0 not found: ID does not exist" Nov 03 18:40:03.784710 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.784668 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:40:03.788527 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.788503 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:40:03.822369 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822342 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:40:03.822726 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822677 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="init-config-reloader" Nov 03 18:40:03.822817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822723 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="init-config-reloader" Nov 03 18:40:03.822817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822745 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy" Nov 03 18:40:03.822817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822754 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy" Nov 03 18:40:03.822817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822770 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="config-reloader" Nov 03 18:40:03.822817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822778 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="config-reloader" Nov 03 18:40:03.822817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822798 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-metric" Nov 03 18:40:03.822817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822805 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-metric" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822830 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="prom-label-proxy" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822838 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="prom-label-proxy" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822869 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-web" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822878 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-web" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822889 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="alertmanager" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822897 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="alertmanager" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.822991 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.823003 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="config-reloader" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.823015 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-metric" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.823029 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="prom-label-proxy" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.823039 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="kube-rbac-proxy-web" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.823046 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="alertmanager" Nov 03 18:40:03.823164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.823057 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" containerName="init-config-reloader" Nov 03 18:40:03.827043 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.827024 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868262 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868236 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868361 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868270 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/171f7ba5-c3b8-4865-82cd-783b05e3028e-config-out\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868361 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868290 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868486 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868373 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171f7ba5-c3b8-4865-82cd-783b05e3028e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868486 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868459 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868579 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868503 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/171f7ba5-c3b8-4865-82cd-783b05e3028e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868615 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868582 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/171f7ba5-c3b8-4865-82cd-783b05e3028e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868615 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868607 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171f7ba5-c3b8-4865-82cd-783b05e3028e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868720 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868645 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868720 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868700 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rs9\" (UniqueName: \"kubernetes.io/projected/171f7ba5-c3b8-4865-82cd-783b05e3028e-kube-api-access-w6rs9\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868819 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868803 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868884 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868836 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-web-config\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.868884 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.868868 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-config-volume\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.876459 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876443 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 18:40:03.876660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876630 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 18:40:03.876912 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876847 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 18:40:03.877003 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876901 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 18:40:03.877003 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876877 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 18:40:03.877003 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876892 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 18:40:03.877003 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876870 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 18:40:03.877003 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.876980 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 18:40:03.877398 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.877386 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jk4jl\"" Nov 03 18:40:03.878145 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.878129 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 18:40:03.923532 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.923512 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:40:03.969572 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969547 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969572 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969574 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rs9\" (UniqueName: \"kubernetes.io/projected/171f7ba5-c3b8-4865-82cd-783b05e3028e-kube-api-access-w6rs9\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969597 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969612 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-web-config\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969628 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-config-volume\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969728 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969774 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/171f7ba5-c3b8-4865-82cd-783b05e3028e-config-out\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969801 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969823 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171f7ba5-c3b8-4865-82cd-783b05e3028e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.969940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969852 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.970140 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969983 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/171f7ba5-c3b8-4865-82cd-783b05e3028e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.970140 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.969984 5988 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 18:40:03.970140 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.970036 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/171f7ba5-c3b8-4865-82cd-783b05e3028e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.970140 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.970067 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171f7ba5-c3b8-4865-82cd-783b05e3028e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.970811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.970786 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171f7ba5-c3b8-4865-82cd-783b05e3028e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.970966 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.970932 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/171f7ba5-c3b8-4865-82cd-783b05e3028e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.971071 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.970879 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/171f7ba5-c3b8-4865-82cd-783b05e3028e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.972801 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.972770 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-web-config\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.972932 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.972897 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.973016 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.972925 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.973077 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.973064 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/171f7ba5-c3b8-4865-82cd-783b05e3028e-config-out\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.973208 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.973183 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.973290 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.973268 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/171f7ba5-c3b8-4865-82cd-783b05e3028e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.973337 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.973313 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.974321 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.974304 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-config-volume\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:03.974360 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:03.974319 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/171f7ba5-c3b8-4865-82cd-783b05e3028e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:04.026965 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:04.026941 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rs9\" (UniqueName: \"kubernetes.io/projected/171f7ba5-c3b8-4865-82cd-783b05e3028e-kube-api-access-w6rs9\") pod \"alertmanager-main-0\" (UID: \"171f7ba5-c3b8-4865-82cd-783b05e3028e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:04.147362 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:04.147300 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 18:40:04.566495 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:04.566469 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 18:40:04.569737 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:40:04.569678 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171f7ba5_c3b8_4865_82cd_783b05e3028e.slice/crio-29bf82a6fd1a6d4694eb76900475dcfe0a9d93c1dcd702871d7f017cfb844722 WatchSource:0}: Error finding container 29bf82a6fd1a6d4694eb76900475dcfe0a9d93c1dcd702871d7f017cfb844722: Status 404 returned error can't find the container with id 29bf82a6fd1a6d4694eb76900475dcfe0a9d93c1dcd702871d7f017cfb844722 Nov 03 18:40:04.674127 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:04.674102 5988 generic.go:356] "Generic (PLEG): container finished" podID="171f7ba5-c3b8-4865-82cd-783b05e3028e" containerID="f8a35005307462034f477aa728253b6c5aaf33b714bc83c06d9472d10b5c7704" exitCode=0 Nov 03 18:40:04.674237 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:04.674190 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerDied","Data":"f8a35005307462034f477aa728253b6c5aaf33b714bc83c06d9472d10b5c7704"} Nov 03 18:40:04.674290 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:04.674276 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerStarted","Data":"29bf82a6fd1a6d4694eb76900475dcfe0a9d93c1dcd702871d7f017cfb844722"} Nov 03 18:40:05.274808 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.274778 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4e2269-545e-4c5d-b5f9-3e337a0fa843" path="/var/lib/kubelet/pods/ce4e2269-545e-4c5d-b5f9-3e337a0fa843/volumes" Nov 03 18:40:05.682371 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.682333 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerStarted","Data":"4875e5ff8dc8a5cdab973d99e3820f92bc6e2d820af8c973993a879bb735e2ef"} Nov 03 18:40:05.682758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.682410 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerStarted","Data":"3ff5079fd5713c8a199df0bb710be618800a03457c3416ffe422e0349e12bf7b"} Nov 03 18:40:05.682758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.682446 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerStarted","Data":"0dd5af8822a3551e942bf4d954b597963743dcfb2a465a49687fee18bd3742f6"} Nov 03 18:40:05.682758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.682473 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerStarted","Data":"6a142643a45fd4de7532cb46eb591d6bbac0e9ca26e6ecd065eb9ee99043029c"} Nov 03 18:40:05.682758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.682504 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerStarted","Data":"b43477b5a5d80177a4499650bd986222be41d3d37a47e1ebd986f8d5ab2af5e3"} Nov 03 18:40:05.682758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.682540 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"171f7ba5-c3b8-4865-82cd-783b05e3028e","Type":"ContainerStarted","Data":"c45735264029d5cd97ecd6228d2a92289d6e657bdc531d57e80ee0a44bbcee84"} Nov 03 18:40:05.805768 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:05.805718 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.797345085 podStartE2EDuration="2.805706692s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.361607ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:05.804678877 +0000 UTC m=+80.686094980" watchObservedRunningTime="2026-11-03 18:40:05.805706692 +0000 UTC m=+80.687122795" Nov 03 18:40:07.288597 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:07.288562 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:07.572287 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:07.572257 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:08.021889 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.021791 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:40:08.022178 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.022144 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="prometheus" containerID="cri-o://60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" gracePeriod=600 Nov 03 18:40:08.022325 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.022206 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" gracePeriod=600 Nov 03 18:40:08.022325 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.022231 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-web" containerID="cri-o://161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" gracePeriod=600 Nov 03 18:40:08.022325 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.022270 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="thanos-sidecar" containerID="cri-o://a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" gracePeriod=600 Nov 03 18:40:08.022484 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.022288 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="config-reloader" containerID="cri-o://39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" gracePeriod=600 Nov 03 18:40:08.022484 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.022241 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy" containerID="cri-o://b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" gracePeriod=600 Nov 03 18:40:08.291696 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.291662 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:08.406811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406783 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.406943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406822 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.406943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406841 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.406943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406865 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.406943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406894 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.406943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406914 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.406943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406938 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406966 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.406992 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407047 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407073 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407105 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407141 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407171 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407206 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407232 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407207 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:08.407716 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407243 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407716 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407278 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407716 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407316 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:08.407716 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407326 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407716 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407396 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407716 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407451 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\") pod \"74be8f9c-362e-4589-b78f-d72f46a83ebc\" (UID: \"74be8f9c-362e-4589-b78f-d72f46a83ebc\") " Nov 03 18:40:08.407716 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407629 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:08.408089 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407807 5988 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-metrics-client-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.408089 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407826 5988 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.408089 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.407842 5988 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.408089 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.408032 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:08.409042 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.408460 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:08.409042 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.408881 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:08.409743 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.409703 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:08.409743 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.409723 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.410064 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.410022 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:08.410361 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.410239 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:08.410361 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.410327 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.410902 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.410840 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:40:08.411457 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.411414 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.411736 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.411694 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.411828 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.411799 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.411903 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.411878 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.412397 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.412376 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "kube-api-access-smrrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:40:08.412479 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.412461 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.412549 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.412530 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.422950 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.422924 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config" pod "74be8f9c-362e-4589-b78f-d72f46a83ebc" (UID: "74be8f9c-362e-4589-b78f-d72f46a83ebc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:08.508820 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508791 5988 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508820 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508820 5988 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508832 5988 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-grpc-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508842 5988 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-tls-assets\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508852 5988 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-web-config\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508860 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smrrx\" (UniqueName: \"kubernetes.io/projected/74be8f9c-362e-4589-b78f-d72f46a83ebc-kube-api-access-smrrx\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508869 5988 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508878 5988 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-config\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508887 5988 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-db\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508909 5988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508918 5988 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508927 5988 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74be8f9c-362e-4589-b78f-d72f46a83ebc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508936 5988 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-kube-rbac-proxy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508945 5988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508954 5988 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74be8f9c-362e-4589-b78f-d72f46a83ebc-config-out\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508963 5988 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-metrics-client-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.508969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.508974 5988 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74be8f9c-362e-4589-b78f-d72f46a83ebc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:08.700609 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700575 5988 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" exitCode=0 Nov 03 18:40:08.700609 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700598 5988 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" exitCode=0 Nov 03 18:40:08.700609 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700607 5988 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" exitCode=0 Nov 03 18:40:08.700609 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700615 5988 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" exitCode=0 Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700621 5988 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" exitCode=0 Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700627 5988 generic.go:356] "Generic (PLEG): container finished" podID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" exitCode=0 Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700672 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700668 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700786 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700830 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700849 5988 scope.go:122] "RemoveContainer" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700873 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700912 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} Nov 03 18:40:08.700983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700952 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} Nov 03 18:40:08.701497 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.700992 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74be8f9c-362e-4589-b78f-d72f46a83ebc","Type":"ContainerDied","Data":"68f415841edd579944524ac5bebc240734d5bdff6f2920f87eedb3230e587478"} Nov 03 18:40:08.708896 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.708878 5988 scope.go:122] "RemoveContainer" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.715491 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.715474 5988 scope.go:122] "RemoveContainer" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.722392 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.722371 5988 scope.go:122] "RemoveContainer" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.728843 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.728817 5988 scope.go:122] "RemoveContainer" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.735120 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.735105 5988 scope.go:122] "RemoveContainer" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.741875 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.741859 5988 scope.go:122] "RemoveContainer" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.748286 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.748267 5988 scope.go:122] "RemoveContainer" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.748533 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:08.748515 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": container with ID starting with 66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff not found: ID does not exist" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.748589 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.748540 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} err="failed to get container status \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": rpc error: code = NotFound desc = could not find container \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": container with ID starting with 66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff not found: ID does not exist" Nov 03 18:40:08.748589 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.748558 5988 scope.go:122] "RemoveContainer" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.748814 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:08.748795 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": container with ID starting with b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9 not found: ID does not exist" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.748859 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.748819 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} err="failed to get container status \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": rpc error: code = NotFound desc = could not find container \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": container with ID starting with b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9 not found: ID does not exist" Nov 03 18:40:08.748859 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.748835 5988 scope.go:122] "RemoveContainer" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.749039 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:08.749022 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": container with ID starting with 161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05 not found: ID does not exist" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.749074 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.749043 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} err="failed to get container status \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": rpc error: code = NotFound desc = could not find container \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": container with ID starting with 161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05 not found: ID does not exist" Nov 03 18:40:08.749074 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.749056 5988 scope.go:122] "RemoveContainer" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.749295 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:08.749281 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": container with ID starting with a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326 not found: ID does not exist" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.749333 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.749299 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} err="failed to get container status \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": rpc error: code = NotFound desc = could not find container \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": container with ID starting with a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326 not found: ID does not exist" Nov 03 18:40:08.749333 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.749309 5988 scope.go:122] "RemoveContainer" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.749518 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:08.749502 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": container with ID starting with 39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593 not found: ID does not exist" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.749589 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.749522 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} err="failed to get container status \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": rpc error: code = NotFound desc = could not find container \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": container with ID starting with 39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593 not found: ID does not exist" Nov 03 18:40:08.749589 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.749533 5988 scope.go:122] "RemoveContainer" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.749999 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:08.749979 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": container with ID starting with 60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42 not found: ID does not exist" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.750042 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750006 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} err="failed to get container status \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": rpc error: code = NotFound desc = could not find container \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": container with ID starting with 60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42 not found: ID does not exist" Nov 03 18:40:08.750042 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750025 5988 scope.go:122] "RemoveContainer" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.750356 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:08.750337 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": container with ID starting with 652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8 not found: ID does not exist" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.750403 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750361 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8"} err="failed to get container status \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": rpc error: code = NotFound desc = could not find container \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": container with ID starting with 652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8 not found: ID does not exist" Nov 03 18:40:08.750403 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750377 5988 scope.go:122] "RemoveContainer" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.750639 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750611 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} err="failed to get container status \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": rpc error: code = NotFound desc = could not find container \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": container with ID starting with 66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff not found: ID does not exist" Nov 03 18:40:08.750729 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750641 5988 scope.go:122] "RemoveContainer" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.750927 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750905 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} err="failed to get container status \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": rpc error: code = NotFound desc = could not find container \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": container with ID starting with b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9 not found: ID does not exist" Nov 03 18:40:08.750984 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.750928 5988 scope.go:122] "RemoveContainer" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.751153 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751135 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} err="failed to get container status \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": rpc error: code = NotFound desc = could not find container \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": container with ID starting with 161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05 not found: ID does not exist" Nov 03 18:40:08.751200 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751153 5988 scope.go:122] "RemoveContainer" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.751382 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751366 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} err="failed to get container status \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": rpc error: code = NotFound desc = could not find container \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": container with ID starting with a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326 not found: ID does not exist" Nov 03 18:40:08.751424 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751383 5988 scope.go:122] "RemoveContainer" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.751626 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751602 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} err="failed to get container status \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": rpc error: code = NotFound desc = could not find container \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": container with ID starting with 39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593 not found: ID does not exist" Nov 03 18:40:08.751626 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751622 5988 scope.go:122] "RemoveContainer" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.751894 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751875 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} err="failed to get container status \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": rpc error: code = NotFound desc = could not find container \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": container with ID starting with 60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42 not found: ID does not exist" Nov 03 18:40:08.751954 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.751895 5988 scope.go:122] "RemoveContainer" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.752112 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752092 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8"} err="failed to get container status \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": rpc error: code = NotFound desc = could not find container \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": container with ID starting with 652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8 not found: ID does not exist" Nov 03 18:40:08.752160 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752112 5988 scope.go:122] "RemoveContainer" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.752347 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752332 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} err="failed to get container status \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": rpc error: code = NotFound desc = could not find container \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": container with ID starting with 66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff not found: ID does not exist" Nov 03 18:40:08.752389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752347 5988 scope.go:122] "RemoveContainer" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.752547 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752529 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} err="failed to get container status \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": rpc error: code = NotFound desc = could not find container \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": container with ID starting with b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9 not found: ID does not exist" Nov 03 18:40:08.752587 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752547 5988 scope.go:122] "RemoveContainer" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.752752 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752736 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} err="failed to get container status \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": rpc error: code = NotFound desc = could not find container \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": container with ID starting with 161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05 not found: ID does not exist" Nov 03 18:40:08.752793 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752753 5988 scope.go:122] "RemoveContainer" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.752954 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752938 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} err="failed to get container status \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": rpc error: code = NotFound desc = could not find container \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": container with ID starting with a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326 not found: ID does not exist" Nov 03 18:40:08.752991 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.752954 5988 scope.go:122] "RemoveContainer" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.753148 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753134 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} err="failed to get container status \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": rpc error: code = NotFound desc = could not find container \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": container with ID starting with 39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593 not found: ID does not exist" Nov 03 18:40:08.753148 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753148 5988 scope.go:122] "RemoveContainer" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.753321 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753306 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} err="failed to get container status \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": rpc error: code = NotFound desc = could not find container \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": container with ID starting with 60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42 not found: ID does not exist" Nov 03 18:40:08.753370 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753320 5988 scope.go:122] "RemoveContainer" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.753510 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753494 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8"} err="failed to get container status \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": rpc error: code = NotFound desc = could not find container \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": container with ID starting with 652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8 not found: ID does not exist" Nov 03 18:40:08.753552 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753511 5988 scope.go:122] "RemoveContainer" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.753777 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753755 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} err="failed to get container status \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": rpc error: code = NotFound desc = could not find container \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": container with ID starting with 66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff not found: ID does not exist" Nov 03 18:40:08.753821 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.753777 5988 scope.go:122] "RemoveContainer" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.754053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754037 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} err="failed to get container status \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": rpc error: code = NotFound desc = could not find container \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": container with ID starting with b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9 not found: ID does not exist" Nov 03 18:40:08.754102 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754053 5988 scope.go:122] "RemoveContainer" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.754265 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754247 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} err="failed to get container status \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": rpc error: code = NotFound desc = could not find container \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": container with ID starting with 161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05 not found: ID does not exist" Nov 03 18:40:08.754305 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754265 5988 scope.go:122] "RemoveContainer" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.754481 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754459 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} err="failed to get container status \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": rpc error: code = NotFound desc = could not find container \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": container with ID starting with a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326 not found: ID does not exist" Nov 03 18:40:08.754481 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754478 5988 scope.go:122] "RemoveContainer" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.754721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754704 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} err="failed to get container status \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": rpc error: code = NotFound desc = could not find container \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": container with ID starting with 39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593 not found: ID does not exist" Nov 03 18:40:08.754721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754721 5988 scope.go:122] "RemoveContainer" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.754958 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754937 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} err="failed to get container status \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": rpc error: code = NotFound desc = could not find container \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": container with ID starting with 60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42 not found: ID does not exist" Nov 03 18:40:08.755005 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.754965 5988 scope.go:122] "RemoveContainer" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.755171 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755153 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8"} err="failed to get container status \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": rpc error: code = NotFound desc = could not find container \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": container with ID starting with 652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8 not found: ID does not exist" Nov 03 18:40:08.755171 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755169 5988 scope.go:122] "RemoveContainer" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.755413 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755394 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} err="failed to get container status \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": rpc error: code = NotFound desc = could not find container \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": container with ID starting with 66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff not found: ID does not exist" Nov 03 18:40:08.755453 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755414 5988 scope.go:122] "RemoveContainer" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.755622 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755606 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} err="failed to get container status \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": rpc error: code = NotFound desc = could not find container \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": container with ID starting with b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9 not found: ID does not exist" Nov 03 18:40:08.755662 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755622 5988 scope.go:122] "RemoveContainer" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.755851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755833 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} err="failed to get container status \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": rpc error: code = NotFound desc = could not find container \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": container with ID starting with 161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05 not found: ID does not exist" Nov 03 18:40:08.755888 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.755853 5988 scope.go:122] "RemoveContainer" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.756055 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756026 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} err="failed to get container status \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": rpc error: code = NotFound desc = could not find container \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": container with ID starting with a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326 not found: ID does not exist" Nov 03 18:40:08.756055 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756049 5988 scope.go:122] "RemoveContainer" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.756300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756283 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} err="failed to get container status \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": rpc error: code = NotFound desc = could not find container \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": container with ID starting with 39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593 not found: ID does not exist" Nov 03 18:40:08.756363 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756300 5988 scope.go:122] "RemoveContainer" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.756525 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756509 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} err="failed to get container status \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": rpc error: code = NotFound desc = could not find container \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": container with ID starting with 60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42 not found: ID does not exist" Nov 03 18:40:08.756525 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756525 5988 scope.go:122] "RemoveContainer" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.756758 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756739 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8"} err="failed to get container status \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": rpc error: code = NotFound desc = could not find container \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": container with ID starting with 652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8 not found: ID does not exist" Nov 03 18:40:08.756819 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756759 5988 scope.go:122] "RemoveContainer" containerID="66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff" Nov 03 18:40:08.756988 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756967 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff"} err="failed to get container status \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": rpc error: code = NotFound desc = could not find container \"66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff\": container with ID starting with 66b34db9cfadd8ce3aabac55f65da9f300c25e619a22ec3d821e6f8bd3d789ff not found: ID does not exist" Nov 03 18:40:08.757037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.756988 5988 scope.go:122] "RemoveContainer" containerID="b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9" Nov 03 18:40:08.757189 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757171 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9"} err="failed to get container status \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": rpc error: code = NotFound desc = could not find container \"b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9\": container with ID starting with b56b9b7ca20ec81a1fa679fdbd1f7f0d77624cea7b45ab3f13b78cd2ac641ba9 not found: ID does not exist" Nov 03 18:40:08.757257 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757190 5988 scope.go:122] "RemoveContainer" containerID="161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05" Nov 03 18:40:08.757430 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757409 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05"} err="failed to get container status \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": rpc error: code = NotFound desc = could not find container \"161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05\": container with ID starting with 161c4c4255cf5bb4edad28f4eeba33409b31fba6b10a7c5d6825ee6bf74b6b05 not found: ID does not exist" Nov 03 18:40:08.757430 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757429 5988 scope.go:122] "RemoveContainer" containerID="a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326" Nov 03 18:40:08.757616 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757599 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326"} err="failed to get container status \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": rpc error: code = NotFound desc = could not find container \"a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326\": container with ID starting with a990c760a993846e2c29f05c2a19b7f81d3d83c0f4d69e517ee8c5fc856da326 not found: ID does not exist" Nov 03 18:40:08.757693 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757617 5988 scope.go:122] "RemoveContainer" containerID="39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593" Nov 03 18:40:08.757838 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757821 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593"} err="failed to get container status \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": rpc error: code = NotFound desc = could not find container \"39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593\": container with ID starting with 39097cd66e5acb62b1a423227fce558ba01bfc8d5a5a7b1dfc5dce6ad4fc2593 not found: ID does not exist" Nov 03 18:40:08.757838 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.757837 5988 scope.go:122] "RemoveContainer" containerID="60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42" Nov 03 18:40:08.758118 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.758098 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42"} err="failed to get container status \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": rpc error: code = NotFound desc = could not find container \"60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42\": container with ID starting with 60c8ed7304970ad29bb14682011cf549800ca44846918105dbf353f0f8e99a42 not found: ID does not exist" Nov 03 18:40:08.758201 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.758119 5988 scope.go:122] "RemoveContainer" containerID="652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8" Nov 03 18:40:08.758353 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.758338 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8"} err="failed to get container status \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": rpc error: code = NotFound desc = could not find container \"652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8\": container with ID starting with 652fe6647a51f9d63db079df255f5fd1fe602d045fcf7d8beb5dbc0c126310f8 not found: ID does not exist" Nov 03 18:40:08.859718 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.859693 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:40:08.906135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.906108 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:40:08.906737 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.906712 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:40:08.907135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907100 5988 cpu_manager.go:414] "containerMap: removing container" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-thanos" Nov 03 18:40:08.907135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907118 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-thanos" Nov 03 18:40:08.907135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907128 5988 cpu_manager.go:414] "containerMap: removing container" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="config-reloader" Nov 03 18:40:08.907135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907134 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="config-reloader" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907143 5988 cpu_manager.go:414] "containerMap: removing container" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="init-config-reloader" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907148 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="init-config-reloader" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907180 5988 cpu_manager.go:414] "containerMap: removing container" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-web" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907186 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-web" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907194 5988 cpu_manager.go:414] "containerMap: removing container" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="thanos-sidecar" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907199 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="thanos-sidecar" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907206 5988 cpu_manager.go:414] "containerMap: removing container" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907210 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907223 5988 cpu_manager.go:414] "containerMap: removing container" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="prometheus" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907228 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="prometheus" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907278 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="init-config-reloader" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907288 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-web" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907293 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="thanos-sidecar" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907299 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy" Nov 03 18:40:08.907300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907306 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="prometheus" Nov 03 18:40:08.907977 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907314 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="kube-rbac-proxy-thanos" Nov 03 18:40:08.907977 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.907319 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" containerName="config-reloader" Nov 03 18:40:08.912166 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.912149 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:08.962058 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.961986 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 18:40:08.962058 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.961988 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 18:40:08.962058 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.962011 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 18:40:08.962426 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.962401 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 18:40:08.962426 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.962404 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 18:40:08.962802 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.962760 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 18:40:08.962802 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.962779 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 18:40:08.962997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.962883 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9c3kkmaf6dql\"" Nov 03 18:40:08.962997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.962903 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 18:40:08.963097 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.963021 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 18:40:08.963097 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.963033 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 18:40:08.963097 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.963066 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-d6l69\"" Nov 03 18:40:08.963097 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:08.963079 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 18:40:09.009623 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.009601 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 18:40:09.011199 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.011179 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 18:40:09.011470 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.011453 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 18:40:09.011985 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.011967 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:40:09.012922 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.012905 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.012987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.012933 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.012987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.012961 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013070 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013056 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013105 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013087 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/56d46121-2a72-4eb8-bbd3-f548a2daac7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013138 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013105 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013138 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013131 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013197 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013166 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013228 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013198 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013260 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013242 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013292 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013273 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013322 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013300 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013360 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013348 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013395 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013372 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013395 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013389 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/56d46121-2a72-4eb8-bbd3-f548a2daac7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013453 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013428 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013493 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013480 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zhf\" (UniqueName: \"kubernetes.io/projected/56d46121-2a72-4eb8-bbd3-f548a2daac7d-kube-api-access-f6zhf\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013527 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013507 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-config\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013559 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013528 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.013559 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.013546 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.114816 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114694 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.114987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114854 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zhf\" (UniqueName: \"kubernetes.io/projected/56d46121-2a72-4eb8-bbd3-f548a2daac7d-kube-api-access-f6zhf\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.114987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114886 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-config\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.114987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114909 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.114987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114928 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.114987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114945 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.114987 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114969 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.114992 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115021 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115057 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/56d46121-2a72-4eb8-bbd3-f548a2daac7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115085 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115125 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115151 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115181 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115217 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.115303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115269 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.116183 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115314 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.116183 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115345 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.116183 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115374 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.116183 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115406 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/56d46121-2a72-4eb8-bbd3-f548a2daac7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.116183 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.115731 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.116486 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.116271 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.116611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.116582 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.117024 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.117001 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.117195 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.117168 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.117567 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.117543 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.117711 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.117590 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.119735 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.119484 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/56d46121-2a72-4eb8-bbd3-f548a2daac7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.119735 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.119719 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/56d46121-2a72-4eb8-bbd3-f548a2daac7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.120504 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.120404 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/56d46121-2a72-4eb8-bbd3-f548a2daac7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.120504 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.120430 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.120880 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.120836 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.120965 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.120941 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.121387 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.121366 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.122518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.122488 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.123378 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.123355 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.123473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.123412 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.123473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.123437 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-config\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.124125 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.124101 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/56d46121-2a72-4eb8-bbd3-f548a2daac7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.172735 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.172712 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zhf\" (UniqueName: \"kubernetes.io/projected/56d46121-2a72-4eb8-bbd3-f548a2daac7d-kube-api-access-f6zhf\") pod \"prometheus-k8s-0\" (UID: \"56d46121-2a72-4eb8-bbd3-f548a2daac7d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.231183 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.231122 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:09.273977 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.273949 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74be8f9c-362e-4589-b78f-d72f46a83ebc" path="/var/lib/kubelet/pods/74be8f9c-362e-4589-b78f-d72f46a83ebc/volumes" Nov 03 18:40:09.656078 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.656014 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 18:40:09.659613 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:40:09.659587 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d46121_2a72_4eb8_bbd3_f548a2daac7d.slice/crio-e5d9175d5902875d99ff47c84c27e1271629e4dee8665bd5ce4d1c06dd6e2c77 WatchSource:0}: Error finding container e5d9175d5902875d99ff47c84c27e1271629e4dee8665bd5ce4d1c06dd6e2c77: Status 404 returned error can't find the container with id e5d9175d5902875d99ff47c84c27e1271629e4dee8665bd5ce4d1c06dd6e2c77 Nov 03 18:40:09.705103 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:09.705078 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerStarted","Data":"e5d9175d5902875d99ff47c84c27e1271629e4dee8665bd5ce4d1c06dd6e2c77"} Nov 03 18:40:10.711319 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:10.711282 5988 generic.go:356] "Generic (PLEG): container finished" podID="56d46121-2a72-4eb8-bbd3-f548a2daac7d" containerID="e0bbf77c34bbd23862a44d4967e13008f9912283e086f463f1f9a258cb19191b" exitCode=0 Nov 03 18:40:10.711319 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:10.711321 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerDied","Data":"e0bbf77c34bbd23862a44d4967e13008f9912283e086f463f1f9a258cb19191b"} Nov 03 18:40:11.718272 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:11.718236 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerStarted","Data":"6b07304a4e9680022a4c1d6b80d291684e9aaa735baa3ae3604781fb44ea2ce5"} Nov 03 18:40:11.718756 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:11.718299 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerStarted","Data":"b051e742df038668b7fbe77c32cb148bd61513ff2f370a2006cf7990e97e92d7"} Nov 03 18:40:11.718756 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:11.718325 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerStarted","Data":"2e5542b9b586ac9ec96956b087a89bb2cb834cebedc249b99be85543c8982f6b"} Nov 03 18:40:11.718756 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:11.718354 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerStarted","Data":"4cdb8072d455a76864aeaf859351200127d5c045aa9481160e548b6c11b8ed1e"} Nov 03 18:40:11.718756 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:11.718377 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerStarted","Data":"c7ecf655bd68ed135a09fd9539a27aff853883755089d58f49bc9d428971a8b1"} Nov 03 18:40:11.718756 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:11.718399 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"56d46121-2a72-4eb8-bbd3-f548a2daac7d","Type":"ContainerStarted","Data":"068cebb7fda5500aa23fa8af168dc88200184f0fd8d2cde812fa953355ec5f69"} Nov 03 18:40:11.843992 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:11.843934 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.829120498 podStartE2EDuration="3.843917354s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.796856ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:11.842525382 +0000 UTC m=+86.723941483" watchObservedRunningTime="2026-11-03 18:40:11.843917354 +0000 UTC m=+86.725333456" Nov 03 18:40:14.231825 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:14.231788 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:40:15.745057 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:15.745016 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:15.948351 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:15.948318 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:16.161435 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:16.161403 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:25.934891 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:25.934801 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:40:40.878752 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:40.878715 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:45.129874 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:45.129819 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:49.431609 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:49.431546 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:49.639784 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:40:49.639727 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:55.934636 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:40:55.934595 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:41:09.232056 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:41:09.232025 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:41:09.248835 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:41:09.248807 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:41:09.950745 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:41:09.950718 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 18:41:25.935152 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:41:25.935114 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:41:41.116060 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:41:41.116023 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:41:55.935604 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:41:55.935514 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:42:20.450566 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:42:20.450530 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:22.118705 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:42:22.118653 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:25.936193 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:42:25.936152 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:42:41.460971 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:42:41.460905 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:55.934505 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:42:55.934471 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:43:23.116383 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:23.116347 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:23.318201 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:23.318164 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:25.935546 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:43:25.935460 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:43:29.056167 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:29.056089 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:29.327124 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:29.327085 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:30.796265 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:30.796204 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:31.005608 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:31.005572 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:32.208404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:32.208324 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:36.722092 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:36.722054 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:36.941385 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:36.941292 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:38.092775 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:38.092737 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:38.322043 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:38.321961 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:38.528392 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:38.528295 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:41.821620 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:41.821536 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:42.176036 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:43:42.176001 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:55.935017 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:43:55.934976 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:44:25.935624 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:44:25.935581 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:44:29.447558 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:29.447504 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:29.660358 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:29.660323 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:30.296090 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.296050 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:30.505372 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.505335 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b"] Nov 03 18:44:30.508134 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.508117 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.561551 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.561478 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 18:44:30.561551 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.561524 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 18:44:30.561917 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.561901 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jdkcr\"" Nov 03 18:44:30.562308 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.562295 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 18:44:30.608409 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.608378 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 18:44:30.608409 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.608382 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 18:44:30.618173 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.618144 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b"] Nov 03 18:44:30.645315 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.645290 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.645449 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.645348 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee301c84-d21d-44a5-8e28-b132c396a6a1-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.645449 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.645430 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.645521 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.645454 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee301c84-d21d-44a5-8e28-b132c396a6a1-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.645555 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.645524 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr42z\" (UniqueName: \"kubernetes.io/projected/ee301c84-d21d-44a5-8e28-b132c396a6a1-kube-api-access-cr42z\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.645622 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.645607 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.676164 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.676134 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:30.746337 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.746310 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee301c84-d21d-44a5-8e28-b132c396a6a1-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.746506 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.746348 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.746506 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.746369 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee301c84-d21d-44a5-8e28-b132c396a6a1-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.746506 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.746490 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr42z\" (UniqueName: \"kubernetes.io/projected/ee301c84-d21d-44a5-8e28-b132c396a6a1-kube-api-access-cr42z\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.746707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.746581 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.746707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.746624 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.746799 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.746762 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ee301c84-d21d-44a5-8e28-b132c396a6a1-manifests\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.748782 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.748753 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee301c84-d21d-44a5-8e28-b132c396a6a1-tmp\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.748986 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.748969 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.749029 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.749011 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.749125 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.749105 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee301c84-d21d-44a5-8e28-b132c396a6a1-metrics-cert\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.804464 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.804433 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr42z\" (UniqueName: \"kubernetes.io/projected/ee301c84-d21d-44a5-8e28-b132c396a6a1-kube-api-access-cr42z\") pod \"opendatahub-operator-controller-manager-7b94cbc577-cjb2b\" (UID: \"ee301c84-d21d-44a5-8e28-b132c396a6a1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.821329 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.821306 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:30.883128 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:30.883085 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:31.088461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:31.088383 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:31.299130 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:31.298961 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b"] Nov 03 18:44:31.302004 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:44:31.301978 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee301c84_d21d_44a5_8e28_b132c396a6a1.slice/crio-9275165552e3331e09af43d6e72d5f9575db2797a88b79548fb9d327e9f79e3d WatchSource:0}: Error finding container 9275165552e3331e09af43d6e72d5f9575db2797a88b79548fb9d327e9f79e3d: Status 404 returned error can't find the container with id 9275165552e3331e09af43d6e72d5f9575db2797a88b79548fb9d327e9f79e3d Nov 03 18:44:31.303787 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:31.303771 5988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:44:31.670865 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:31.670828 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" event={"ID":"ee301c84-d21d-44a5-8e28-b132c396a6a1","Type":"ContainerStarted","Data":"9275165552e3331e09af43d6e72d5f9575db2797a88b79548fb9d327e9f79e3d"} Nov 03 18:44:31.795983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:31.795940 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:32.814569 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:32.814534 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:33.018992 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:33.018960 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:34.017983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.017946 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:34.225404 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.225371 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:34.464678 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.464643 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj"] Nov 03 18:44:34.470954 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.470930 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:34.567330 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.567277 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 18:44:34.567521 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.567361 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 18:44:34.567521 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.567402 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7qlhr\"" Nov 03 18:44:34.568441 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.568412 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj"] Nov 03 18:44:34.579990 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.579967 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pch46\" (UniqueName: \"kubernetes.io/projected/400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f-kube-api-access-pch46\") pod \"jobset-operator-674cd59dc5-dx8lj\" (UID: \"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:34.580155 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.580109 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f-tmp\") pod \"jobset-operator-674cd59dc5-dx8lj\" (UID: \"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:34.681663 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.681630 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f-tmp\") pod \"jobset-operator-674cd59dc5-dx8lj\" (UID: \"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:34.681864 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.681723 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pch46\" (UniqueName: \"kubernetes.io/projected/400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f-kube-api-access-pch46\") pod \"jobset-operator-674cd59dc5-dx8lj\" (UID: \"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:34.682098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.682072 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f-tmp\") pod \"jobset-operator-674cd59dc5-dx8lj\" (UID: \"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:34.750852 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.750772 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pch46\" (UniqueName: \"kubernetes.io/projected/400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f-kube-api-access-pch46\") pod \"jobset-operator-674cd59dc5-dx8lj\" (UID: \"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:34.793541 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:34.793499 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" Nov 03 18:44:35.190775 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:35.190740 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj"] Nov 03 18:44:35.196013 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:44:35.195970 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod400ff61b_d1ef_4cd5_9fc5_2e3744eeb79f.slice/crio-9511ef21220f2585666378b3100f5c8d6827ae09b2ac75bd0768a3a8be4182a6 WatchSource:0}: Error finding container 9511ef21220f2585666378b3100f5c8d6827ae09b2ac75bd0768a3a8be4182a6: Status 404 returned error can't find the container with id 9511ef21220f2585666378b3100f5c8d6827ae09b2ac75bd0768a3a8be4182a6 Nov 03 18:44:35.691177 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:35.691136 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" event={"ID":"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f","Type":"ContainerStarted","Data":"9511ef21220f2585666378b3100f5c8d6827ae09b2ac75bd0768a3a8be4182a6"} Nov 03 18:44:36.696786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:36.696745 5988 generic.go:356] "Generic (PLEG): container finished" podID="ee301c84-d21d-44a5-8e28-b132c396a6a1" containerID="e5e6db857e35a2ddd74543b645cfd175b1f9d6b5b0c57d3833bbfbb98a1fd2da" exitCode=0 Nov 03 18:44:36.697170 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:36.696835 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" event={"ID":"ee301c84-d21d-44a5-8e28-b132c396a6a1","Type":"ContainerDied","Data":"e5e6db857e35a2ddd74543b645cfd175b1f9d6b5b0c57d3833bbfbb98a1fd2da"} Nov 03 18:44:37.702602 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:37.702564 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" event={"ID":"ee301c84-d21d-44a5-8e28-b132c396a6a1","Type":"ContainerStarted","Data":"6e570e14494af043d835aa0ee725f88711ef0678308f5f97f2259460b55b7b47"} Nov 03 18:44:37.703034 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:37.702728 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:37.925514 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:37.925445 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" podStartSLOduration=2.236468775 podStartE2EDuration="7.925429036s" totalImagesPullingTime="5.201308952s" totalInitContainerRuntime="487.651309ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:37.879239932 +0000 UTC m=+352.760656033" watchObservedRunningTime="2026-11-03 18:44:37.925429036 +0000 UTC m=+352.806845135" Nov 03 18:44:39.415218 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:39.415186 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:39.634858 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:39.634819 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:39.712038 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:39.711948 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" event={"ID":"400ff61b-d1ef-4cd5-9fc5-2e3744eeb79f","Type":"ContainerStarted","Data":"184fa6be4567aa095ded02031517bf65a121ecfbda81ed1329182bac483d1d54"} Nov 03 18:44:39.821856 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:39.821795 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dx8lj" podStartSLOduration=1.555971439 podStartE2EDuration="5.82177837s" totalImagesPullingTime="4.265806931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:39.821421934 +0000 UTC m=+354.702838037" watchObservedRunningTime="2026-11-03 18:44:39.82177837 +0000 UTC m=+354.703194472" Nov 03 18:44:40.009790 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:40.009669 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:40.226032 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:40.225984 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:41.384468 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:41.384435 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:42.194659 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:42.194621 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:42.627070 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:42.627036 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:46.788108 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:46.788060 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:47.596961 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:47.596927 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:48.708303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:48.708275 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7b94cbc577-cjb2b" Nov 03 18:44:49.845292 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:49.845251 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:50.070618 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:50.070586 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:51.216990 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:51.216953 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:52.008201 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:52.008168 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:52.801901 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:52.801869 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:53.630732 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:53.630677 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:54.083227 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:54.083184 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:55.314824 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:55.314793 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:55.533816 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:55.533776 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:55.935593 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:44:55.935551 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:44:56.158699 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:56.158654 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:56.439933 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:56.439901 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:57.036941 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.036909 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-247bb"] Nov 03 18:44:57.039358 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.039333 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.083060 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.083023 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79aabaf9-34bb-44c0-a979-d83452d3b6a8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-247bb\" (UID: \"79aabaf9-34bb-44c0-a979-d83452d3b6a8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.083217 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.083145 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68m7q\" (UniqueName: \"kubernetes.io/projected/79aabaf9-34bb-44c0-a979-d83452d3b6a8-kube-api-access-68m7q\") pod \"cert-manager-webhook-5dbbbf75cc-247bb\" (UID: \"79aabaf9-34bb-44c0-a979-d83452d3b6a8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.089932 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.089903 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 18:44:57.136378 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.136340 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 18:44:57.136544 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.136451 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8hwkw\"" Nov 03 18:44:57.137288 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.137265 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-247bb"] Nov 03 18:44:57.184081 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.184045 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79aabaf9-34bb-44c0-a979-d83452d3b6a8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-247bb\" (UID: \"79aabaf9-34bb-44c0-a979-d83452d3b6a8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.184246 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.184118 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68m7q\" (UniqueName: \"kubernetes.io/projected/79aabaf9-34bb-44c0-a979-d83452d3b6a8-kube-api-access-68m7q\") pod \"cert-manager-webhook-5dbbbf75cc-247bb\" (UID: \"79aabaf9-34bb-44c0-a979-d83452d3b6a8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.244286 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.244247 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/79aabaf9-34bb-44c0-a979-d83452d3b6a8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-247bb\" (UID: \"79aabaf9-34bb-44c0-a979-d83452d3b6a8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.244438 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.244257 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68m7q\" (UniqueName: \"kubernetes.io/projected/79aabaf9-34bb-44c0-a979-d83452d3b6a8-kube-api-access-68m7q\") pod \"cert-manager-webhook-5dbbbf75cc-247bb\" (UID: \"79aabaf9-34bb-44c0-a979-d83452d3b6a8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.350054 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.350018 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:44:57.715778 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.715649 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-247bb"] Nov 03 18:44:57.718707 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:44:57.718654 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79aabaf9_34bb_44c0_a979_d83452d3b6a8.slice/crio-43a2c24c13bf91ae0ecb547814704ff71480281fc31f2cb69bf20057230ef5b7 WatchSource:0}: Error finding container 43a2c24c13bf91ae0ecb547814704ff71480281fc31f2cb69bf20057230ef5b7: Status 404 returned error can't find the container with id 43a2c24c13bf91ae0ecb547814704ff71480281fc31f2cb69bf20057230ef5b7 Nov 03 18:44:57.800112 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:57.800079 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" event={"ID":"79aabaf9-34bb-44c0-a979-d83452d3b6a8","Type":"ContainerStarted","Data":"43a2c24c13bf91ae0ecb547814704ff71480281fc31f2cb69bf20057230ef5b7"} Nov 03 18:44:58.431360 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:58.431321 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:58.793201 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:58.793113 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:58.918140 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:58.918104 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh"] Nov 03 18:44:58.921360 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:58.921341 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:58.971939 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:58.971909 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6dsp4\"" Nov 03 18:44:59.001434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.001401 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7ae3323-3c99-43cb-8c85-ff23dfd874f8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-99fqh\" (UID: \"b7ae3323-3c99-43cb-8c85-ff23dfd874f8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:59.001434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.001435 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxz4\" (UniqueName: \"kubernetes.io/projected/b7ae3323-3c99-43cb-8c85-ff23dfd874f8-kube-api-access-2nxz4\") pod \"cert-manager-cainjector-5c57bbd4fc-99fqh\" (UID: \"b7ae3323-3c99-43cb-8c85-ff23dfd874f8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:59.015132 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.015102 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:44:59.019664 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.019644 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh"] Nov 03 18:44:59.102969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.102932 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7ae3323-3c99-43cb-8c85-ff23dfd874f8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-99fqh\" (UID: \"b7ae3323-3c99-43cb-8c85-ff23dfd874f8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:59.102969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.102971 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxz4\" (UniqueName: \"kubernetes.io/projected/b7ae3323-3c99-43cb-8c85-ff23dfd874f8-kube-api-access-2nxz4\") pod \"cert-manager-cainjector-5c57bbd4fc-99fqh\" (UID: \"b7ae3323-3c99-43cb-8c85-ff23dfd874f8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:59.160885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.160846 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7ae3323-3c99-43cb-8c85-ff23dfd874f8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-99fqh\" (UID: \"b7ae3323-3c99-43cb-8c85-ff23dfd874f8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:59.161007 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.160956 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxz4\" (UniqueName: \"kubernetes.io/projected/b7ae3323-3c99-43cb-8c85-ff23dfd874f8-kube-api-access-2nxz4\") pod \"cert-manager-cainjector-5c57bbd4fc-99fqh\" (UID: \"b7ae3323-3c99-43cb-8c85-ff23dfd874f8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:59.231980 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.231945 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" Nov 03 18:44:59.600187 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.600154 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh"] Nov 03 18:44:59.605397 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:44:59.605364 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7ae3323_3c99_43cb_8c85_ff23dfd874f8.slice/crio-d80f8500d89cb419ef6118c69502fa69757ed201e036467f1306e7a93023fe81 WatchSource:0}: Error finding container d80f8500d89cb419ef6118c69502fa69757ed201e036467f1306e7a93023fe81: Status 404 returned error can't find the container with id d80f8500d89cb419ef6118c69502fa69757ed201e036467f1306e7a93023fe81 Nov 03 18:44:59.812384 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:44:59.812339 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" event={"ID":"b7ae3323-3c99-43cb-8c85-ff23dfd874f8","Type":"ContainerStarted","Data":"d80f8500d89cb419ef6118c69502fa69757ed201e036467f1306e7a93023fe81"} Nov 03 18:45:01.784861 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:01.784793 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:02.079078 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:02.079039 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:02.342662 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:02.342587 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:02.708362 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:02.708268 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:02.829958 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:02.829920 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" event={"ID":"b7ae3323-3c99-43cb-8c85-ff23dfd874f8","Type":"ContainerStarted","Data":"e01094e730301def67572cd88b9369e945984d90beae1913c247e64e434c2c5e"} Nov 03 18:45:02.831642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:02.831611 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" event={"ID":"79aabaf9-34bb-44c0-a979-d83452d3b6a8","Type":"ContainerStarted","Data":"f2fce326c828d954ffba98870465ab887b26e16d4a874b83ab12ddb304afed12"} Nov 03 18:45:02.831828 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:02.831813 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:45:02.952059 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:02.951997 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-99fqh" podStartSLOduration=2.037648559 podStartE2EDuration="4.951977577s" totalImagesPullingTime="2.914329018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:02.950269644 +0000 UTC m=+377.831685739" watchObservedRunningTime="2026-11-03 18:45:02.951977577 +0000 UTC m=+377.833393680" Nov 03 18:45:03.076152 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:03.076099 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" podStartSLOduration=2.692115194 podStartE2EDuration="7.076085458s" totalImagesPullingTime="4.383970264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:03.075009092 +0000 UTC m=+377.956425194" watchObservedRunningTime="2026-11-03 18:45:03.076085458 +0000 UTC m=+377.957501559" Nov 03 18:45:04.016053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.016018 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:04.271386 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.271302 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:04.417794 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.417765 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b8f876b79-hw55c"] Nov 03 18:45:04.421354 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.421329 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.475714 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.475659 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 18:45:04.475895 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.475763 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 18:45:04.475895 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.475762 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 18:45:04.475895 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.475764 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 18:45:04.490528 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.490499 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:04.522641 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.522565 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kgrlf\"" Nov 03 18:45:04.523569 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.523547 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8f876b79-hw55c"] Nov 03 18:45:04.561024 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.560987 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/707684d4-a619-4320-b02f-177056b4065b-tmp\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.561203 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.561072 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/707684d4-a619-4320-b02f-177056b4065b-tls-certs\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.561203 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.561116 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r589k\" (UniqueName: \"kubernetes.io/projected/707684d4-a619-4320-b02f-177056b4065b-kube-api-access-r589k\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.662463 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.662425 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/707684d4-a619-4320-b02f-177056b4065b-tmp\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.662636 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.662491 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/707684d4-a619-4320-b02f-177056b4065b-tls-certs\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.662636 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.662520 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r589k\" (UniqueName: \"kubernetes.io/projected/707684d4-a619-4320-b02f-177056b4065b-kube-api-access-r589k\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.664727 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.664703 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/707684d4-a619-4320-b02f-177056b4065b-tmp\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.664972 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.664951 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/707684d4-a619-4320-b02f-177056b4065b-tls-certs\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.706055 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.706021 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:04.720767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.720740 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r589k\" (UniqueName: \"kubernetes.io/projected/707684d4-a619-4320-b02f-177056b4065b-kube-api-access-r589k\") pod \"kube-auth-proxy-b8f876b79-hw55c\" (UID: \"707684d4-a619-4320-b02f-177056b4065b\") " pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.734302 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.734271 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" Nov 03 18:45:04.923676 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:04.923629 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:05.122123 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:05.122049 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8f876b79-hw55c"] Nov 03 18:45:05.124657 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:45:05.124633 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707684d4_a619_4320_b02f_177056b4065b.slice/crio-d8b79f811a838871785b02d11a9f4171673eba769da2c33ebf644c2aba784716 WatchSource:0}: Error finding container d8b79f811a838871785b02d11a9f4171673eba769da2c33ebf644c2aba784716: Status 404 returned error can't find the container with id d8b79f811a838871785b02d11a9f4171673eba769da2c33ebf644c2aba784716 Nov 03 18:45:05.143594 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:05.143566 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:05.357110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:05.357078 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:05.853137 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:05.853102 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" event={"ID":"707684d4-a619-4320-b02f-177056b4065b","Type":"ContainerStarted","Data":"d8b79f811a838871785b02d11a9f4171673eba769da2c33ebf644c2aba784716"} Nov 03 18:45:06.535789 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.535750 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbb99"] Nov 03 18:45:06.539693 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.539662 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:06.581176 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.581145 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2438a02d-260b-45a5-880e-6a8a76e24bd4-bound-sa-token\") pod \"cert-manager-597b95cb84-rbb99\" (UID: \"2438a02d-260b-45a5-880e-6a8a76e24bd4\") " pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:06.581394 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.581187 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tcsf\" (UniqueName: \"kubernetes.io/projected/2438a02d-260b-45a5-880e-6a8a76e24bd4-kube-api-access-5tcsf\") pod \"cert-manager-597b95cb84-rbb99\" (UID: \"2438a02d-260b-45a5-880e-6a8a76e24bd4\") " pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:06.638551 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.638502 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbb99"] Nov 03 18:45:06.681820 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.681777 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2438a02d-260b-45a5-880e-6a8a76e24bd4-bound-sa-token\") pod \"cert-manager-597b95cb84-rbb99\" (UID: \"2438a02d-260b-45a5-880e-6a8a76e24bd4\") " pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:06.682023 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.681828 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tcsf\" (UniqueName: \"kubernetes.io/projected/2438a02d-260b-45a5-880e-6a8a76e24bd4-kube-api-access-5tcsf\") pod \"cert-manager-597b95cb84-rbb99\" (UID: \"2438a02d-260b-45a5-880e-6a8a76e24bd4\") " pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:06.690195 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.690158 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6cjhg\"" Nov 03 18:45:06.748179 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.748141 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:06.750322 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.750294 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tcsf\" (UniqueName: \"kubernetes.io/projected/2438a02d-260b-45a5-880e-6a8a76e24bd4-kube-api-access-5tcsf\") pod \"cert-manager-597b95cb84-rbb99\" (UID: \"2438a02d-260b-45a5-880e-6a8a76e24bd4\") " pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:06.750465 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.750342 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2438a02d-260b-45a5-880e-6a8a76e24bd4-bound-sa-token\") pod \"cert-manager-597b95cb84-rbb99\" (UID: \"2438a02d-260b-45a5-880e-6a8a76e24bd4\") " pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:06.855986 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:06.855957 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rbb99" Nov 03 18:45:07.232642 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:07.232559 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbb99"] Nov 03 18:45:07.236643 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:45:07.236607 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2438a02d_260b_45a5_880e_6a8a76e24bd4.slice/crio-92f1dfb71dd8fa06746561250953e77a873ff4e5b748bf1dba51bcf00f13a835 WatchSource:0}: Error finding container 92f1dfb71dd8fa06746561250953e77a873ff4e5b748bf1dba51bcf00f13a835: Status 404 returned error can't find the container with id 92f1dfb71dd8fa06746561250953e77a873ff4e5b748bf1dba51bcf00f13a835 Nov 03 18:45:07.863577 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:07.863542 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rbb99" event={"ID":"2438a02d-260b-45a5-880e-6a8a76e24bd4","Type":"ContainerStarted","Data":"f7c3adb31318c939a90e380a93b2542d3a28c42d7a5918486275d65b3033100b"} Nov 03 18:45:07.863577 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:07.863585 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rbb99" event={"ID":"2438a02d-260b-45a5-880e-6a8a76e24bd4","Type":"ContainerStarted","Data":"92f1dfb71dd8fa06746561250953e77a873ff4e5b748bf1dba51bcf00f13a835"} Nov 03 18:45:07.992887 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:07.992817 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rbb99" podStartSLOduration=1.9928015860000001 podStartE2EDuration="1.992801586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:07.991792167 +0000 UTC m=+382.873208281" watchObservedRunningTime="2026-11-03 18:45:07.992801586 +0000 UTC m=+382.874217687" Nov 03 18:45:08.839219 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:08.839185 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" Nov 03 18:45:09.813334 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:09.813301 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:09.874319 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:09.874275 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" event={"ID":"707684d4-a619-4320-b02f-177056b4065b","Type":"ContainerStarted","Data":"ca567238cc0eee86f7f39a7f4a2de2569415f47fc87f1e72a3be5c6cea5dac89"} Nov 03 18:45:09.988605 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:09.988547 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b8f876b79-hw55c" podStartSLOduration=2.252440817 podStartE2EDuration="5.988529937s" totalImagesPullingTime="3.73608912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:09.986785052 +0000 UTC m=+384.868201155" watchObservedRunningTime="2026-11-03 18:45:09.988529937 +0000 UTC m=+384.869946075" Nov 03 18:45:10.021917 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:10.021885 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:11.429764 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.429730 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:11.447731 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.447694 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vmswn"] Nov 03 18:45:11.451374 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.451351 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.505978 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.505942 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 18:45:11.527464 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.527430 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4cjt\" (UniqueName: \"kubernetes.io/projected/8a2aee57-6585-4bec-8d4e-9ef351f151e7-kube-api-access-m4cjt\") pod \"limitador-limitador-74dfcfb894-vmswn\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.527614 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.527488 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a2aee57-6585-4bec-8d4e-9ef351f151e7-config-file\") pod \"limitador-limitador-74dfcfb894-vmswn\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.552517 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.552487 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4z4fj\"" Nov 03 18:45:11.552596 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.552539 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 18:45:11.553692 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.553664 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vmswn"] Nov 03 18:45:11.605823 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.605794 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 18:45:11.628228 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.628198 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4cjt\" (UniqueName: \"kubernetes.io/projected/8a2aee57-6585-4bec-8d4e-9ef351f151e7-kube-api-access-m4cjt\") pod \"limitador-limitador-74dfcfb894-vmswn\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.628341 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.628266 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a2aee57-6585-4bec-8d4e-9ef351f151e7-config-file\") pod \"limitador-limitador-74dfcfb894-vmswn\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.628951 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.628933 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a2aee57-6585-4bec-8d4e-9ef351f151e7-config-file\") pod \"limitador-limitador-74dfcfb894-vmswn\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.686059 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.685986 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4cjt\" (UniqueName: \"kubernetes.io/projected/8a2aee57-6585-4bec-8d4e-9ef351f151e7-kube-api-access-m4cjt\") pod \"limitador-limitador-74dfcfb894-vmswn\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.699508 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.699465 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vmswn"] Nov 03 18:45:11.764528 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.764497 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:11.797747 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.797704 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:11.998462 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:11.998383 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:12.134297 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:12.134137 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vmswn"] Nov 03 18:45:12.137091 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:45:12.137061 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2aee57_6585_4bec_8d4e_9ef351f151e7.slice/crio-6a0dddfea38d206719bfa877a3e84fa1e3b8cf6991328ddbcf59e15bf659fae4 WatchSource:0}: Error finding container 6a0dddfea38d206719bfa877a3e84fa1e3b8cf6991328ddbcf59e15bf659fae4: Status 404 returned error can't find the container with id 6a0dddfea38d206719bfa877a3e84fa1e3b8cf6991328ddbcf59e15bf659fae4 Nov 03 18:45:12.402514 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:12.402470 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:12.889010 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:12.888975 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" event={"ID":"8a2aee57-6585-4bec-8d4e-9ef351f151e7","Type":"ContainerStarted","Data":"6a0dddfea38d206719bfa877a3e84fa1e3b8cf6991328ddbcf59e15bf659fae4"} Nov 03 18:45:13.178638 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:13.178555 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:13.389873 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:13.389798 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:15.606428 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:15.606393 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:15.812111 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:15.812076 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:15.917766 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:15.917703 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:45:15.921221 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:15.921198 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:15.984525 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:15.984501 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:45:16.013873 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.013850 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:16.076185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.076155 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsxt\" (UniqueName: \"kubernetes.io/projected/71691d5d-452d-4976-998f-715fb0207c0f-kube-api-access-jrsxt\") pod \"limitador-limitador-7c9b87d574-sclr2\" (UID: \"71691d5d-452d-4976-998f-715fb0207c0f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:16.076296 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.076277 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71691d5d-452d-4976-998f-715fb0207c0f-config-file\") pod \"limitador-limitador-7c9b87d574-sclr2\" (UID: \"71691d5d-452d-4976-998f-715fb0207c0f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:16.177434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.177349 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsxt\" (UniqueName: \"kubernetes.io/projected/71691d5d-452d-4976-998f-715fb0207c0f-kube-api-access-jrsxt\") pod \"limitador-limitador-7c9b87d574-sclr2\" (UID: \"71691d5d-452d-4976-998f-715fb0207c0f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:16.177434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.177430 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71691d5d-452d-4976-998f-715fb0207c0f-config-file\") pod \"limitador-limitador-7c9b87d574-sclr2\" (UID: \"71691d5d-452d-4976-998f-715fb0207c0f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:16.177993 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.177975 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71691d5d-452d-4976-998f-715fb0207c0f-config-file\") pod \"limitador-limitador-7c9b87d574-sclr2\" (UID: \"71691d5d-452d-4976-998f-715fb0207c0f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:16.226496 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.226469 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:45:16.231183 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.231161 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:16.234587 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.234568 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsxt\" (UniqueName: \"kubernetes.io/projected/71691d5d-452d-4976-998f-715fb0207c0f-kube-api-access-jrsxt\") pod \"limitador-limitador-7c9b87d574-sclr2\" (UID: \"71691d5d-452d-4976-998f-715fb0207c0f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:16.235360 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.235344 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:16.450201 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.450126 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:16.599379 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.599353 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:45:16.602198 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:45:16.602174 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71691d5d_452d_4976_998f_715fb0207c0f.slice/crio-d747ba0c84305ad74fd1f83409e9f672fd462f7e7f3793853972fdf6a727d864 WatchSource:0}: Error finding container d747ba0c84305ad74fd1f83409e9f672fd462f7e7f3793853972fdf6a727d864: Status 404 returned error can't find the container with id d747ba0c84305ad74fd1f83409e9f672fd462f7e7f3793853972fdf6a727d864 Nov 03 18:45:16.910314 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.910282 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" event={"ID":"8a2aee57-6585-4bec-8d4e-9ef351f151e7","Type":"ContainerStarted","Data":"1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07"} Nov 03 18:45:16.910775 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.910401 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:16.916615 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.912790 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" event={"ID":"71691d5d-452d-4976-998f-715fb0207c0f","Type":"ContainerStarted","Data":"7776ddf45216647ae4da97916e3e6924c462766f6e29baa708a43219a8125b3b"} Nov 03 18:45:16.916615 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.912888 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" event={"ID":"71691d5d-452d-4976-998f-715fb0207c0f","Type":"ContainerStarted","Data":"d747ba0c84305ad74fd1f83409e9f672fd462f7e7f3793853972fdf6a727d864"} Nov 03 18:45:16.916615 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:16.913142 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:17.019146 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:17.019096 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" podStartSLOduration=2.189092565 podStartE2EDuration="6.01908347s" totalImagesPullingTime="3.829990905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:17.018330532 +0000 UTC m=+391.899746633" watchObservedRunningTime="2026-11-03 18:45:17.01908347 +0000 UTC m=+391.900499565" Nov 03 18:45:17.128671 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:17.128621 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" podStartSLOduration=2.1286024709999998 podStartE2EDuration="2.128602471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:17.127613932 +0000 UTC m=+392.009030033" watchObservedRunningTime="2026-11-03 18:45:17.128602471 +0000 UTC m=+392.010018572" Nov 03 18:45:21.130475 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:21.130440 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:21.332340 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:21.332305 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:22.871148 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:22.871110 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:25.903422 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:25.903386 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:25.935614 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:45:25.935580 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:45:26.126512 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:26.126482 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:26.829774 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:26.829741 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:27.040820 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:27.040783 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:27.918163 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:27.918133 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:27.918327 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:27.918199 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-sclr2" Nov 03 18:45:28.165913 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.165879 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vmswn"] Nov 03 18:45:28.166303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.166072 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" podUID="8a2aee57-6585-4bec-8d4e-9ef351f151e7" containerName="limitador" containerID="cri-o://1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07" gracePeriod=30 Nov 03 18:45:28.709106 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.709082 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:28.884095 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.884061 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2aee57-6585-4bec-8d4e-9ef351f151e7-config-file\" (UniqueName: \"kubernetes.io/configmap/8a2aee57-6585-4bec-8d4e-9ef351f151e7-config-file\") pod \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " Nov 03 18:45:28.884263 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.884201 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2aee57-6585-4bec-8d4e-9ef351f151e7-kube-api-access-m4cjt\" (UniqueName: \"kubernetes.io/projected/8a2aee57-6585-4bec-8d4e-9ef351f151e7-kube-api-access-m4cjt\") pod \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\" (UID: \"8a2aee57-6585-4bec-8d4e-9ef351f151e7\") " Nov 03 18:45:28.884371 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.884348 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2aee57-6585-4bec-8d4e-9ef351f151e7-config-file" pod "8a2aee57-6585-4bec-8d4e-9ef351f151e7" (UID: "8a2aee57-6585-4bec-8d4e-9ef351f151e7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:28.886291 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.886268 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2aee57-6585-4bec-8d4e-9ef351f151e7-kube-api-access-m4cjt" pod "8a2aee57-6585-4bec-8d4e-9ef351f151e7" (UID: "8a2aee57-6585-4bec-8d4e-9ef351f151e7"). InnerVolumeSpecName "kube-api-access-m4cjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:28.967334 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.967300 5988 generic.go:356] "Generic (PLEG): container finished" podID="8a2aee57-6585-4bec-8d4e-9ef351f151e7" containerID="1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07" exitCode=0 Nov 03 18:45:28.967461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.967340 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" event={"ID":"8a2aee57-6585-4bec-8d4e-9ef351f151e7","Type":"ContainerDied","Data":"1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07"} Nov 03 18:45:28.967461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.967348 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" Nov 03 18:45:28.967461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.967414 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vmswn" event={"ID":"8a2aee57-6585-4bec-8d4e-9ef351f151e7","Type":"ContainerDied","Data":"6a0dddfea38d206719bfa877a3e84fa1e3b8cf6991328ddbcf59e15bf659fae4"} Nov 03 18:45:28.967461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.967438 5988 scope.go:122] "RemoveContainer" containerID="1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07" Nov 03 18:45:28.975719 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.975675 5988 scope.go:122] "RemoveContainer" containerID="1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07" Nov 03 18:45:28.976016 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:45:28.975998 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07\": container with ID starting with 1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07 not found: ID does not exist" containerID="1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07" Nov 03 18:45:28.976063 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.976025 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07"} err="failed to get container status \"1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07\": rpc error: code = NotFound desc = could not find container \"1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07\": container with ID starting with 1595dde5317fe85630e88ab1d0df1c5c0f44ca3b61e2a56c065e26b411dc7f07 not found: ID does not exist" Nov 03 18:45:28.985422 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.985401 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4cjt\" (UniqueName: \"kubernetes.io/projected/8a2aee57-6585-4bec-8d4e-9ef351f151e7-kube-api-access-m4cjt\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:28.985505 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:28.985424 5988 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a2aee57-6585-4bec-8d4e-9ef351f151e7-config-file\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:29.120175 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:29.120148 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vmswn"] Nov 03 18:45:29.123928 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:29.123908 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vmswn"] Nov 03 18:45:29.271771 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:29.271708 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2aee57-6585-4bec-8d4e-9ef351f151e7" path="/var/lib/kubelet/pods/8a2aee57-6585-4bec-8d4e-9ef351f151e7/volumes" Nov 03 18:45:29.489949 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:29.489893 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:29.730141 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:29.730114 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:34.282972 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:34.282940 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:34.486112 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:34.486080 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:34.867956 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:34.867921 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:35.069960 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.069922 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:35.304306 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.304229 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:35.465762 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.465730 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq"] Nov 03 18:45:35.466113 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.466100 5988 cpu_manager.go:414] "containerMap: removing container" podUID="8a2aee57-6585-4bec-8d4e-9ef351f151e7" containerName="limitador" Nov 03 18:45:35.466155 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.466114 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2aee57-6585-4bec-8d4e-9ef351f151e7" containerName="limitador" Nov 03 18:45:35.466188 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.466175 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2aee57-6585-4bec-8d4e-9ef351f151e7" containerName="limitador" Nov 03 18:45:35.473153 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.472715 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:35.509394 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.509367 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:35.523050 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.523024 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 18:45:35.569088 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.569069 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:45:35.569357 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.569341 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6z2lm\"" Nov 03 18:45:35.569427 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.569417 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:45:35.570340 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.570323 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq"] Nov 03 18:45:35.640736 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.640707 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47sz\" (UniqueName: \"kubernetes.io/projected/f7d65397-5369-4595-baa2-84ee90ea69b2-kube-api-access-f47sz\") pod \"postgres-7cdbf6dc44-r8xrq\" (UID: \"f7d65397-5369-4595-baa2-84ee90ea69b2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:35.640858 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.640753 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f7d65397-5369-4595-baa2-84ee90ea69b2-data\") pod \"postgres-7cdbf6dc44-r8xrq\" (UID: \"f7d65397-5369-4595-baa2-84ee90ea69b2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:35.722704 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.722649 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:35.741861 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.741836 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47sz\" (UniqueName: \"kubernetes.io/projected/f7d65397-5369-4595-baa2-84ee90ea69b2-kube-api-access-f47sz\") pod \"postgres-7cdbf6dc44-r8xrq\" (UID: \"f7d65397-5369-4595-baa2-84ee90ea69b2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:35.741988 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.741879 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f7d65397-5369-4595-baa2-84ee90ea69b2-data\") pod \"postgres-7cdbf6dc44-r8xrq\" (UID: \"f7d65397-5369-4595-baa2-84ee90ea69b2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:35.742200 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.742185 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f7d65397-5369-4595-baa2-84ee90ea69b2-data\") pod \"postgres-7cdbf6dc44-r8xrq\" (UID: \"f7d65397-5369-4595-baa2-84ee90ea69b2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:35.799623 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.799596 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47sz\" (UniqueName: \"kubernetes.io/projected/f7d65397-5369-4595-baa2-84ee90ea69b2-kube-api-access-f47sz\") pod \"postgres-7cdbf6dc44-r8xrq\" (UID: \"f7d65397-5369-4595-baa2-84ee90ea69b2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:35.937172 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:35.937097 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:36.084581 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:36.084557 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:36.479872 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:45:36.479840 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d65397_5369_4595_baa2_84ee90ea69b2.slice/crio-5ef45297f6bb340b720729a313aad194aae26be9a79c3b4f4a7b6b5c4d6f4caa WatchSource:0}: Error finding container 5ef45297f6bb340b720729a313aad194aae26be9a79c3b4f4a7b6b5c4d6f4caa: Status 404 returned error can't find the container with id 5ef45297f6bb340b720729a313aad194aae26be9a79c3b4f4a7b6b5c4d6f4caa Nov 03 18:45:36.480290 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:36.480088 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq"] Nov 03 18:45:37.018664 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:37.018628 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" event={"ID":"f7d65397-5369-4595-baa2-84ee90ea69b2","Type":"ContainerStarted","Data":"5ef45297f6bb340b720729a313aad194aae26be9a79c3b4f4a7b6b5c4d6f4caa"} Nov 03 18:45:40.744138 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:40.744087 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:40.953134 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:40.953094 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:42.508024 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:42.507974 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:44.084366 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:44.084313 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" event={"ID":"f7d65397-5369-4595-baa2-84ee90ea69b2","Type":"ContainerStarted","Data":"1ea1c73c0c33e28cc6a1bc5176df0f3a8d11992862f053bf8f7568af29393dfc"} Nov 03 18:45:44.084895 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:44.084433 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:44.194528 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:44.194459 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" podStartSLOduration=2.509489297 podStartE2EDuration="9.194440943s" totalImagesPullingTime="6.684951646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:44.193084229 +0000 UTC m=+419.074500332" watchObservedRunningTime="2026-11-03 18:45:44.194440943 +0000 UTC m=+419.075857046" Nov 03 18:45:50.116669 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:50.116640 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-r8xrq" Nov 03 18:45:54.963640 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:54.963609 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:55.168880 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:55.168834 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:55.935316 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:45:55.935280 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:45:57.185142 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:57.185104 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:57.406915 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:57.406856 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:57.619436 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:57.619410 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:45:57.818923 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:45:57.818890 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:02.048964 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:02.048931 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:02.291260 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:02.291222 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:05.013025 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:05.012992 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:05.220128 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:05.220090 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:10.647551 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:10.647509 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:25.935869 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:46:25.935789 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:46:29.435917 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:29.435838 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:29.642432 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:29.642392 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:30.162281 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:30.162253 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:31.928580 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:31.928545 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:33.535632 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:33.535602 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:33.818416 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:33.818386 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:35.697917 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:35.697871 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:35.904737 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:35.904701 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:36.108591 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.108555 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:36.749044 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.749014 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp"] Nov 03 18:46:36.751715 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.751677 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.788019 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.787991 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-maas-api-tls\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.788135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.788041 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-metrics-certs\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.788135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.788121 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdfg\" (UniqueName: \"kubernetes.io/projected/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-kube-api-access-7mdfg\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.788220 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.788165 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-openshift-service-ca\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.847342 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.847312 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\"" Nov 03 18:46:36.847598 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.847445 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 18:46:36.847598 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.847526 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:46:36.848724 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.848678 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp"] Nov 03 18:46:36.889311 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.889286 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-metrics-certs\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.889415 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.889326 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdfg\" (UniqueName: \"kubernetes.io/projected/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-kube-api-access-7mdfg\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.889415 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.889359 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-openshift-service-ca\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.889521 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.889420 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-maas-api-tls\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.890064 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.890036 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-openshift-service-ca\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.891668 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.891644 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-metrics-certs\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.891736 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.891670 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-maas-api-tls\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:36.947511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:36.947483 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdfg\" (UniqueName: \"kubernetes.io/projected/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-kube-api-access-7mdfg\") pod \"maas-api-6d669c7fcd-4r7sp\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:37.063407 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:37.063331 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:37.426261 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:37.426232 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp"] Nov 03 18:46:37.429468 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:46:37.429441 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c52c8a1_188d_465e_b7e0_2722eaa74bb6.slice/crio-744ddd8571d688feff67567189920cf27f97b1aedf771c7c4ecb32873e955267 WatchSource:0}: Error finding container 744ddd8571d688feff67567189920cf27f97b1aedf771c7c4ecb32873e955267: Status 404 returned error can't find the container with id 744ddd8571d688feff67567189920cf27f97b1aedf771c7c4ecb32873e955267 Nov 03 18:46:37.700407 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:37.700332 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:37.905353 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:37.905323 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:38.137270 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.137242 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:38.316498 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.316463 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" event={"ID":"9c52c8a1-188d-465e-b7e0-2722eaa74bb6","Type":"ContainerStarted","Data":"744ddd8571d688feff67567189920cf27f97b1aedf771c7c4ecb32873e955267"} Nov 03 18:46:38.361213 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.361176 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:38.543559 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.543481 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jgk4x"] Nov 03 18:46:38.546001 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.545978 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.565776 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.565747 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:38.595273 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.595254 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 18:46:38.595273 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.595258 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-w8kwn\"" Nov 03 18:46:38.595459 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.595276 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 18:46:38.595459 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.595331 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 18:46:38.600910 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.600890 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jgk4x"] Nov 03 18:46:38.605967 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.605947 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28965a9-7d03-49e4-b4ab-ed87baf626c4-cert\") pod \"kserve-controller-manager-776cc66d47-jgk4x\" (UID: \"e28965a9-7d03-49e4-b4ab-ed87baf626c4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.606088 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.605996 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9qw\" (UniqueName: \"kubernetes.io/projected/e28965a9-7d03-49e4-b4ab-ed87baf626c4-kube-api-access-vf9qw\") pod \"kserve-controller-manager-776cc66d47-jgk4x\" (UID: \"e28965a9-7d03-49e4-b4ab-ed87baf626c4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.657433 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.657399 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927"] Nov 03 18:46:38.661216 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.661190 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.706834 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.706794 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28965a9-7d03-49e4-b4ab-ed87baf626c4-cert\") pod \"kserve-controller-manager-776cc66d47-jgk4x\" (UID: \"e28965a9-7d03-49e4-b4ab-ed87baf626c4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.706976 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.706854 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9qw\" (UniqueName: \"kubernetes.io/projected/e28965a9-7d03-49e4-b4ab-ed87baf626c4-kube-api-access-vf9qw\") pod \"kserve-controller-manager-776cc66d47-jgk4x\" (UID: \"e28965a9-7d03-49e4-b4ab-ed87baf626c4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.706976 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.706949 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4b2\" (UniqueName: \"kubernetes.io/projected/bfa84ad5-9726-44ef-988b-a070dbcc7219-kube-api-access-bw4b2\") pod \"llmisvc-controller-manager-869d5dcc5d-ps927\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.707180 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.706984 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa84ad5-9726-44ef-988b-a070dbcc7219-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ps927\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.709970 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.709945 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e28965a9-7d03-49e4-b4ab-ed87baf626c4-cert\") pod \"kserve-controller-manager-776cc66d47-jgk4x\" (UID: \"e28965a9-7d03-49e4-b4ab-ed87baf626c4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.710728 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.710706 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 18:46:38.710953 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.710828 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-wl5hf\"" Nov 03 18:46:38.712018 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.711978 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927"] Nov 03 18:46:38.775305 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.775234 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:38.777162 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.777131 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5px4m"] Nov 03 18:46:38.778621 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.778596 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9qw\" (UniqueName: \"kubernetes.io/projected/e28965a9-7d03-49e4-b4ab-ed87baf626c4-kube-api-access-vf9qw\") pod \"kserve-controller-manager-776cc66d47-jgk4x\" (UID: \"e28965a9-7d03-49e4-b4ab-ed87baf626c4\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.780514 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.780496 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:38.808106 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.808078 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4b2\" (UniqueName: \"kubernetes.io/projected/bfa84ad5-9726-44ef-988b-a070dbcc7219-kube-api-access-bw4b2\") pod \"llmisvc-controller-manager-869d5dcc5d-ps927\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.808218 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.808133 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa84ad5-9726-44ef-988b-a070dbcc7219-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ps927\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.811356 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.811332 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa84ad5-9726-44ef-988b-a070dbcc7219-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ps927\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.827654 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.827603 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5px4m"] Nov 03 18:46:38.829728 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.829668 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 18:46:38.829842 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.829754 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fcwv8\"" Nov 03 18:46:38.858905 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.858882 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:38.868707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.868658 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4b2\" (UniqueName: \"kubernetes.io/projected/bfa84ad5-9726-44ef-988b-a070dbcc7219-kube-api-access-bw4b2\") pod \"llmisvc-controller-manager-869d5dcc5d-ps927\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.893267 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.893240 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-8rqlm"] Nov 03 18:46:38.911978 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.911934 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnllt\" (UniqueName: \"kubernetes.io/projected/51a5ffea-2b78-499e-b3f3-556157548f4b-kube-api-access-cnllt\") pod \"model-serving-api-75bd89fdfc-5px4m\" (UID: \"51a5ffea-2b78-499e-b3f3-556157548f4b\") " pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:38.912357 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.912039 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a5ffea-2b78-499e-b3f3-556157548f4b-tls-certs\") pod \"model-serving-api-75bd89fdfc-5px4m\" (UID: \"51a5ffea-2b78-499e-b3f3-556157548f4b\") " pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:38.912357 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.912090 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:38.946592 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.946565 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8rqlm"] Nov 03 18:46:38.962664 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.962629 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 18:46:38.963199 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.963173 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wm7r6\"" Nov 03 18:46:38.963738 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.963461 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 18:46:38.977755 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.977714 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:38.984822 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:38.984742 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:39.012946 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.012908 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a5ffea-2b78-499e-b3f3-556157548f4b-tls-certs\") pod \"model-serving-api-75bd89fdfc-5px4m\" (UID: \"51a5ffea-2b78-499e-b3f3-556157548f4b\") " pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:39.013753 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.013073 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-cert\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.013753 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.013111 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-metrics-certs\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.013753 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.013157 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnllt\" (UniqueName: \"kubernetes.io/projected/51a5ffea-2b78-499e-b3f3-556157548f4b-kube-api-access-cnllt\") pod \"model-serving-api-75bd89fdfc-5px4m\" (UID: \"51a5ffea-2b78-499e-b3f3-556157548f4b\") " pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:39.013753 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.013202 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6rf\" (UniqueName: \"kubernetes.io/projected/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-kube-api-access-qf6rf\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.018346 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.018303 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a5ffea-2b78-499e-b3f3-556157548f4b-tls-certs\") pod \"model-serving-api-75bd89fdfc-5px4m\" (UID: \"51a5ffea-2b78-499e-b3f3-556157548f4b\") " pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:39.076892 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.074971 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnllt\" (UniqueName: \"kubernetes.io/projected/51a5ffea-2b78-499e-b3f3-556157548f4b-kube-api-access-cnllt\") pod \"model-serving-api-75bd89fdfc-5px4m\" (UID: \"51a5ffea-2b78-499e-b3f3-556157548f4b\") " pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:39.111874 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.111846 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:39.114199 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.114170 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6rf\" (UniqueName: \"kubernetes.io/projected/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-kube-api-access-qf6rf\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.117317 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.117297 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-cert\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.117443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.117432 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-metrics-certs\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.120266 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.120241 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-cert\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.129935 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.129914 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-metrics-certs\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.173519 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.173484 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6rf\" (UniqueName: \"kubernetes.io/projected/6e6addd9-43a6-4705-9c7b-b9a3e35938a4-kube-api-access-qf6rf\") pod \"odh-model-controller-769875db56-8rqlm\" (UID: \"6e6addd9-43a6-4705-9c7b-b9a3e35938a4\") " pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.257085 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.256929 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jgk4x"] Nov 03 18:46:39.259777 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.259373 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:39.259981 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:46:39.259948 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28965a9_7d03_49e4_b4ab_ed87baf626c4.slice/crio-7fc304be537717bfdfcdca9decce2eb5a179a63742524e548bbc261764310530 WatchSource:0}: Error finding container 7fc304be537717bfdfcdca9decce2eb5a179a63742524e548bbc261764310530: Status 404 returned error can't find the container with id 7fc304be537717bfdfcdca9decce2eb5a179a63742524e548bbc261764310530 Nov 03 18:46:39.332060 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.332023 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" event={"ID":"e28965a9-7d03-49e4-b4ab-ed87baf626c4","Type":"ContainerStarted","Data":"7fc304be537717bfdfcdca9decce2eb5a179a63742524e548bbc261764310530"} Nov 03 18:46:39.366360 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.366331 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927"] Nov 03 18:46:39.372750 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:46:39.372720 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfa84ad5_9726_44ef_988b_a070dbcc7219.slice/crio-72694877feac10592a3215b81daa6dee836c5c277990e7183d14ab6f679a1734 WatchSource:0}: Error finding container 72694877feac10592a3215b81daa6dee836c5c277990e7183d14ab6f679a1734: Status 404 returned error can't find the container with id 72694877feac10592a3215b81daa6dee836c5c277990e7183d14ab6f679a1734 Nov 03 18:46:39.497355 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.497331 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5px4m"] Nov 03 18:46:39.500401 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:46:39.500359 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a5ffea_2b78_499e_b3f3_556157548f4b.slice/crio-ac41553b7f421bc1dba79c211be50c5c8fbf9a935de736db1c0bc0c3ac84f89b WatchSource:0}: Error finding container ac41553b7f421bc1dba79c211be50c5c8fbf9a935de736db1c0bc0c3ac84f89b: Status 404 returned error can't find the container with id ac41553b7f421bc1dba79c211be50c5c8fbf9a935de736db1c0bc0c3ac84f89b Nov 03 18:46:39.650399 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:39.650368 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8rqlm"] Nov 03 18:46:39.654222 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:46:39.654195 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6addd9_43a6_4705_9c7b_b9a3e35938a4.slice/crio-e42a0820adc8d3420ec8314f6555d3ddfc188a4874ef81da6427b1dae9089d92 WatchSource:0}: Error finding container e42a0820adc8d3420ec8314f6555d3ddfc188a4874ef81da6427b1dae9089d92: Status 404 returned error can't find the container with id e42a0820adc8d3420ec8314f6555d3ddfc188a4874ef81da6427b1dae9089d92 Nov 03 18:46:40.078889 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:40.078847 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:40.339939 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:40.339811 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" event={"ID":"51a5ffea-2b78-499e-b3f3-556157548f4b","Type":"ContainerStarted","Data":"ac41553b7f421bc1dba79c211be50c5c8fbf9a935de736db1c0bc0c3ac84f89b"} Nov 03 18:46:40.341138 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:40.341109 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8rqlm" event={"ID":"6e6addd9-43a6-4705-9c7b-b9a3e35938a4","Type":"ContainerStarted","Data":"e42a0820adc8d3420ec8314f6555d3ddfc188a4874ef81da6427b1dae9089d92"} Nov 03 18:46:40.342426 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:40.342390 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" event={"ID":"bfa84ad5-9726-44ef-988b-a070dbcc7219","Type":"ContainerStarted","Data":"72694877feac10592a3215b81daa6dee836c5c277990e7183d14ab6f679a1734"} Nov 03 18:46:41.353525 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:41.352614 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" event={"ID":"9c52c8a1-188d-465e-b7e0-2722eaa74bb6","Type":"ContainerStarted","Data":"21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a"} Nov 03 18:46:41.353525 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:41.353462 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:41.465697 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:41.465628 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" podStartSLOduration=1.960920793 podStartE2EDuration="5.465610653s" totalImagesPullingTime="3.50468986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:41.465128543 +0000 UTC m=+476.346544646" watchObservedRunningTime="2026-11-03 18:46:41.465610653 +0000 UTC m=+476.347026756" Nov 03 18:46:42.363150 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:42.363109 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" event={"ID":"51a5ffea-2b78-499e-b3f3-556157548f4b","Type":"ContainerStarted","Data":"990e9558efc549ec12093771212210afaace07e93bd109ca368e02c36fe99a1d"} Nov 03 18:46:42.363616 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:42.363180 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:42.399049 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:42.399014 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:46:42.505733 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:42.505621 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" podStartSLOduration=1.851604381 podStartE2EDuration="4.505602345s" totalImagesPullingTime="2.653997964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:42.504718918 +0000 UTC m=+477.386135020" watchObservedRunningTime="2026-11-03 18:46:42.505602345 +0000 UTC m=+477.387018448" Nov 03 18:46:42.866051 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:42.866015 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:44.372321 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.372289 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8rqlm" event={"ID":"6e6addd9-43a6-4705-9c7b-b9a3e35938a4","Type":"ContainerStarted","Data":"d89245f640a3850494b1841db51477201165e82c0b47bfb827bc431242d629bb"} Nov 03 18:46:44.372770 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.372386 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:44.373851 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.373826 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" event={"ID":"bfa84ad5-9726-44ef-988b-a070dbcc7219","Type":"ContainerStarted","Data":"8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd"} Nov 03 18:46:44.373974 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.373882 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:46:44.375520 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.375497 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" event={"ID":"e28965a9-7d03-49e4-b4ab-ed87baf626c4","Type":"ContainerStarted","Data":"685b29ec4295c522e889f7b1fdc991aaa03f5045b763e6900c83e8ba33001e40"} Nov 03 18:46:44.481998 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.481950 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-8rqlm" podStartSLOduration=2.312200229 podStartE2EDuration="6.481936957s" totalImagesPullingTime="4.169736728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:44.481077624 +0000 UTC m=+479.362493729" watchObservedRunningTime="2026-11-03 18:46:44.481936957 +0000 UTC m=+479.363353037" Nov 03 18:46:44.590874 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.590818 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" podStartSLOduration=2.517553322 podStartE2EDuration="6.59080446s" totalImagesPullingTime="4.073251138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:44.590135135 +0000 UTC m=+479.471551239" watchObservedRunningTime="2026-11-03 18:46:44.59080446 +0000 UTC m=+479.472220578" Nov 03 18:46:44.699187 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:44.699094 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" podStartSLOduration=2.735568691 podStartE2EDuration="6.69907979s" totalImagesPullingTime="3.963511099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:44.698553541 +0000 UTC m=+479.579969644" watchObservedRunningTime="2026-11-03 18:46:44.69907979 +0000 UTC m=+479.580495923" Nov 03 18:46:45.279582 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:45.279544 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:45.380470 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:45.380430 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:46:45.505794 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:45.505752 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:45.749811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:45.749772 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:45.947997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:45.947960 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:46.156430 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:46.156401 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:46.374406 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:46.374366 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:46.580750 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:46.580718 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:48.372381 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:48.372350 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:46:48.946007 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:48.945973 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:49.191590 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:49.191548 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:46:54.380101 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:54.380074 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-5px4m" Nov 03 18:46:55.382658 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:46:55.382624 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-8rqlm" Nov 03 18:46:55.935706 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:46:55.935655 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:47:01.013003 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.012951 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp"] Nov 03 18:47:01.013536 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.013239 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" podUID="9c52c8a1-188d-465e-b7e0-2722eaa74bb6" containerName="maas-api" containerID="cri-o://21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a" gracePeriod=30 Nov 03 18:47:01.259105 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.259075 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:47:01.328666 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.328637 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-openshift-service-ca\") pod \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " Nov 03 18:47:01.328815 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.328713 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-maas-api-tls\") pod \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " Nov 03 18:47:01.328815 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.328748 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-kube-api-access-7mdfg\" (UniqueName: \"kubernetes.io/projected/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-kube-api-access-7mdfg\") pod \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " Nov 03 18:47:01.328898 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.328824 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-metrics-certs\") pod \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\" (UID: \"9c52c8a1-188d-465e-b7e0-2722eaa74bb6\") " Nov 03 18:47:01.329318 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.329285 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-openshift-service-ca" pod "9c52c8a1-188d-465e-b7e0-2722eaa74bb6" (UID: "9c52c8a1-188d-465e-b7e0-2722eaa74bb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:47:01.331097 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.331073 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-maas-api-tls" pod "9c52c8a1-188d-465e-b7e0-2722eaa74bb6" (UID: "9c52c8a1-188d-465e-b7e0-2722eaa74bb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:47:01.331225 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.331095 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-metrics-certs" pod "9c52c8a1-188d-465e-b7e0-2722eaa74bb6" (UID: "9c52c8a1-188d-465e-b7e0-2722eaa74bb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:47:01.331225 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.331123 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-kube-api-access-7mdfg" pod "9c52c8a1-188d-465e-b7e0-2722eaa74bb6" (UID: "9c52c8a1-188d-465e-b7e0-2722eaa74bb6"). InnerVolumeSpecName "kube-api-access-7mdfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:47:01.430216 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.430179 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mdfg\" (UniqueName: \"kubernetes.io/projected/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-kube-api-access-7mdfg\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:47:01.430216 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.430215 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:47:01.430418 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.430232 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:47:01.430418 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.430246 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c52c8a1-188d-465e-b7e0-2722eaa74bb6-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:47:01.454649 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.454616 5988 generic.go:356] "Generic (PLEG): container finished" podID="9c52c8a1-188d-465e-b7e0-2722eaa74bb6" containerID="21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a" exitCode=0 Nov 03 18:47:01.454828 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.454664 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" Nov 03 18:47:01.454828 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.454651 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" event={"ID":"9c52c8a1-188d-465e-b7e0-2722eaa74bb6","Type":"ContainerDied","Data":"21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a"} Nov 03 18:47:01.454828 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.454812 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp" event={"ID":"9c52c8a1-188d-465e-b7e0-2722eaa74bb6","Type":"ContainerDied","Data":"744ddd8571d688feff67567189920cf27f97b1aedf771c7c4ecb32873e955267"} Nov 03 18:47:01.454930 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.454846 5988 scope.go:122] "RemoveContainer" containerID="21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a" Nov 03 18:47:01.464420 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.464405 5988 scope.go:122] "RemoveContainer" containerID="21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a" Nov 03 18:47:01.464675 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:47:01.464656 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a\": container with ID starting with 21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a not found: ID does not exist" containerID="21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a" Nov 03 18:47:01.464737 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.464705 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a"} err="failed to get container status \"21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a\": rpc error: code = NotFound desc = could not find container \"21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a\": container with ID starting with 21e0669406c9c709a2fbf4fa06cd524472d8fd1dae03c39ce0b7beb8dc27139a not found: ID does not exist" Nov 03 18:47:01.608270 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.608183 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp"] Nov 03 18:47:01.612125 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:01.612101 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-4r7sp"] Nov 03 18:47:03.271773 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:03.271732 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c52c8a1-188d-465e-b7e0-2722eaa74bb6" path="/var/lib/kubelet/pods/9c52c8a1-188d-465e-b7e0-2722eaa74bb6/volumes" Nov 03 18:47:15.382606 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:15.382575 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:47:16.389948 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:16.389910 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jgk4x" Nov 03 18:47:25.935290 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:47:25.935251 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:47:43.214256 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:43.214183 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:47:55.935357 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:47:55.935277 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:47:57.449830 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.449787 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7z57r"] Nov 03 18:47:57.450251 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.450192 5988 cpu_manager.go:414] "containerMap: removing container" podUID="9c52c8a1-188d-465e-b7e0-2722eaa74bb6" containerName="maas-api" Nov 03 18:47:57.450251 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.450204 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c52c8a1-188d-465e-b7e0-2722eaa74bb6" containerName="maas-api" Nov 03 18:47:57.450335 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.450291 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c52c8a1-188d-465e-b7e0-2722eaa74bb6" containerName="maas-api" Nov 03 18:47:57.452591 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.452573 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:57.509222 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.509193 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7z57r"] Nov 03 18:47:57.514186 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.514163 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lsl\" (UniqueName: \"kubernetes.io/projected/e644f9f1-4403-4823-a83a-e1822f286a97-kube-api-access-q2lsl\") pod \"llmisvc-controller-manager-6466c554d8-7z57r\" (UID: \"e644f9f1-4403-4823-a83a-e1822f286a97\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:57.514347 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.514323 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e644f9f1-4403-4823-a83a-e1822f286a97-cert\") pod \"llmisvc-controller-manager-6466c554d8-7z57r\" (UID: \"e644f9f1-4403-4823-a83a-e1822f286a97\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:57.615497 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.615464 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e644f9f1-4403-4823-a83a-e1822f286a97-cert\") pod \"llmisvc-controller-manager-6466c554d8-7z57r\" (UID: \"e644f9f1-4403-4823-a83a-e1822f286a97\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:57.615641 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.615509 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lsl\" (UniqueName: \"kubernetes.io/projected/e644f9f1-4403-4823-a83a-e1822f286a97-kube-api-access-q2lsl\") pod \"llmisvc-controller-manager-6466c554d8-7z57r\" (UID: \"e644f9f1-4403-4823-a83a-e1822f286a97\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:57.617841 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.617818 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e644f9f1-4403-4823-a83a-e1822f286a97-cert\") pod \"llmisvc-controller-manager-6466c554d8-7z57r\" (UID: \"e644f9f1-4403-4823-a83a-e1822f286a97\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:57.673776 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.673747 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lsl\" (UniqueName: \"kubernetes.io/projected/e644f9f1-4403-4823-a83a-e1822f286a97-kube-api-access-q2lsl\") pod \"llmisvc-controller-manager-6466c554d8-7z57r\" (UID: \"e644f9f1-4403-4823-a83a-e1822f286a97\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:57.764625 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:57.764557 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:58.126486 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:58.126454 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7z57r"] Nov 03 18:47:58.129604 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:47:58.129577 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode644f9f1_4403_4823_a83a_e1822f286a97.slice/crio-934d35532da35910742dcf4bf8215af0e10ae7e0db130d209b4f37c4c69839b0 WatchSource:0}: Error finding container 934d35532da35910742dcf4bf8215af0e10ae7e0db130d209b4f37c4c69839b0: Status 404 returned error can't find the container with id 934d35532da35910742dcf4bf8215af0e10ae7e0db130d209b4f37c4c69839b0 Nov 03 18:47:58.723826 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:58.723786 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" event={"ID":"e644f9f1-4403-4823-a83a-e1822f286a97","Type":"ContainerStarted","Data":"934d35532da35910742dcf4bf8215af0e10ae7e0db130d209b4f37c4c69839b0"} Nov 03 18:47:59.729718 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:59.729661 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" event={"ID":"e644f9f1-4403-4823-a83a-e1822f286a97","Type":"ContainerStarted","Data":"b1f02cc801a0bb5dcf2c9e7b86afdd2b681571b776f9fe60434b9d6ba7649a6e"} Nov 03 18:47:59.730111 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:59.729846 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:47:59.858728 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:47:59.858657 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" podStartSLOduration=2.042902814 podStartE2EDuration="2.858641249s" totalImagesPullingTime="815.738435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:59.858438375 +0000 UTC m=+554.739854478" watchObservedRunningTime="2026-11-03 18:47:59.858641249 +0000 UTC m=+554.740057350" Nov 03 18:48:09.532547 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:09.532495 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:20.232528 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:20.232490 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:25.935510 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:48:25.935474 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:48:28.099679 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:28.099643 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:28.354653 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:28.354535 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:30.736765 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:30.736734 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:48:30.923845 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:30.923812 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927"] Nov 03 18:48:30.924034 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:30.924008 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" podUID="bfa84ad5-9726-44ef-988b-a070dbcc7219" containerName="manager" containerID="cri-o://8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd" gracePeriod=30 Nov 03 18:48:31.166900 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.166866 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:48:31.308956 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.308895 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa84ad5-9726-44ef-988b-a070dbcc7219-cert\" (UniqueName: \"kubernetes.io/secret/bfa84ad5-9726-44ef-988b-a070dbcc7219-cert\") pod \"bfa84ad5-9726-44ef-988b-a070dbcc7219\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " Nov 03 18:48:31.309083 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.308956 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa84ad5-9726-44ef-988b-a070dbcc7219-kube-api-access-bw4b2\" (UniqueName: \"kubernetes.io/projected/bfa84ad5-9726-44ef-988b-a070dbcc7219-kube-api-access-bw4b2\") pod \"bfa84ad5-9726-44ef-988b-a070dbcc7219\" (UID: \"bfa84ad5-9726-44ef-988b-a070dbcc7219\") " Nov 03 18:48:31.311004 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.310978 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa84ad5-9726-44ef-988b-a070dbcc7219-cert" pod "bfa84ad5-9726-44ef-988b-a070dbcc7219" (UID: "bfa84ad5-9726-44ef-988b-a070dbcc7219"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:48:31.311110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.311040 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa84ad5-9726-44ef-988b-a070dbcc7219-kube-api-access-bw4b2" pod "bfa84ad5-9726-44ef-988b-a070dbcc7219" (UID: "bfa84ad5-9726-44ef-988b-a070dbcc7219"). InnerVolumeSpecName "kube-api-access-bw4b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:48:31.410271 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.410245 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw4b2\" (UniqueName: \"kubernetes.io/projected/bfa84ad5-9726-44ef-988b-a070dbcc7219-kube-api-access-bw4b2\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:31.410271 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.410269 5988 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa84ad5-9726-44ef-988b-a070dbcc7219-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:48:31.872497 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.872459 5988 generic.go:356] "Generic (PLEG): container finished" podID="bfa84ad5-9726-44ef-988b-a070dbcc7219" containerID="8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd" exitCode=0 Nov 03 18:48:31.873034 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.872511 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" Nov 03 18:48:31.873034 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.872542 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" event={"ID":"bfa84ad5-9726-44ef-988b-a070dbcc7219","Type":"ContainerDied","Data":"8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd"} Nov 03 18:48:31.873034 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.872596 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927" event={"ID":"bfa84ad5-9726-44ef-988b-a070dbcc7219","Type":"ContainerDied","Data":"72694877feac10592a3215b81daa6dee836c5c277990e7183d14ab6f679a1734"} Nov 03 18:48:31.873034 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.872618 5988 scope.go:122] "RemoveContainer" containerID="8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd" Nov 03 18:48:31.882094 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.882076 5988 scope.go:122] "RemoveContainer" containerID="8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd" Nov 03 18:48:31.882358 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:48:31.882340 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd\": container with ID starting with 8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd not found: ID does not exist" containerID="8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd" Nov 03 18:48:31.882407 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.882366 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd"} err="failed to get container status \"8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd\": rpc error: code = NotFound desc = could not find container \"8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd\": container with ID starting with 8a52f26e0538f5656bc524bff05555dfa094a8df90495d67ae00aa4a0e8275dd not found: ID does not exist" Nov 03 18:48:31.959823 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:31.959795 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:32.046364 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:32.046337 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927"] Nov 03 18:48:32.052296 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:32.052274 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ps927"] Nov 03 18:48:32.202014 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:32.201928 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:33.270942 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:33.270912 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa84ad5-9726-44ef-988b-a070dbcc7219" path="/var/lib/kubelet/pods/bfa84ad5-9726-44ef-988b-a070dbcc7219/volumes" Nov 03 18:48:43.593384 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:43.593351 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:53.993962 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:53.993929 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:48:55.934821 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:48:55.934780 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:48:57.060745 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:57.060708 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:48:58.436996 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:48:58.436947 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:02.358547 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:02.358512 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:04.176633 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:04.176602 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:49:06.559016 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:06.558970 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:10.716420 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:10.716388 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:49:14.522191 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:14.522159 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:18.480724 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:18.480644 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:22.054251 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:22.054208 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:24.607937 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:24.607856 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:49:24.962388 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:24.962319 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd"] Nov 03 18:49:24.963017 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:24.962997 5988 cpu_manager.go:414] "containerMap: removing container" podUID="bfa84ad5-9726-44ef-988b-a070dbcc7219" containerName="manager" Nov 03 18:49:24.963068 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:24.963022 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa84ad5-9726-44ef-988b-a070dbcc7219" containerName="manager" Nov 03 18:49:24.967425 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:24.967389 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa84ad5-9726-44ef-988b-a070dbcc7219" containerName="manager" Nov 03 18:49:24.971015 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:24.970987 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.022223 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.022194 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 18:49:25.068586 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.068561 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h6x4t\"" Nov 03 18:49:25.070713 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.070661 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd"] Nov 03 18:49:25.071290 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071269 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csq62\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-kube-api-access-csq62\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071348 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071301 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071348 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071326 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071457 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071403 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b6678b-d97b-47f6-8698-8841a50d93a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071532 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071517 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071591 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071572 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071698 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071659 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071755 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071730 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.071832 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.071818 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.172921 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.172892 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.172921 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.172924 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173139 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.172951 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173139 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.172967 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173139 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.172992 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173139 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173086 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csq62\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-kube-api-access-csq62\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173139 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173114 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173407 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173154 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173407 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173334 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173407 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173362 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b6678b-d97b-47f6-8698-8841a50d93a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173567 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173443 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173805 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173784 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.173894 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.173812 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.174082 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.174062 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b6678b-d97b-47f6-8698-8841a50d93a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.175145 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.175126 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.175504 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.175485 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.231528 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.231464 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.231853 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.231836 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csq62\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-kube-api-access-csq62\") pod \"maas-default-gateway-openshift-default-7854949c9-6t8sd\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.283788 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.283765 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:25.626833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.626802 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:25.698354 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:25.698325 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd"] Nov 03 18:49:25.701258 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:49:25.701230 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b6678b_d97b_47f6_8698_8841a50d93a9.slice/crio-fdb5db8fe00d5aa47a68ccaae22f5ffccc3522ab0fee99ae7eafc5f2b9817088 WatchSource:0}: Error finding container fdb5db8fe00d5aa47a68ccaae22f5ffccc3522ab0fee99ae7eafc5f2b9817088: Status 404 returned error can't find the container with id fdb5db8fe00d5aa47a68ccaae22f5ffccc3522ab0fee99ae7eafc5f2b9817088 Nov 03 18:49:25.938056 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:49:25.937979 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:49:26.126771 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:26.126738 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" event={"ID":"88b6678b-d97b-47f6-8698-8841a50d93a9","Type":"ContainerStarted","Data":"fdb5db8fe00d5aa47a68ccaae22f5ffccc3522ab0fee99ae7eafc5f2b9817088"} Nov 03 18:49:29.733080 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:29.733042 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:49:29.733467 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:29.733128 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:49:29.733467 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:29.733177 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:49:30.148756 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:30.148718 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" event={"ID":"88b6678b-d97b-47f6-8698-8841a50d93a9","Type":"ContainerStarted","Data":"94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd"} Nov 03 18:49:30.284723 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:30.284663 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:30.286070 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:30.286037 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 18:49:30.286229 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:30.286099 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 18:49:30.309224 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:30.309174 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podStartSLOduration=2.280031506 podStartE2EDuration="6.309160158s" totalImagesPullingTime="4.029128652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:30.308240018 +0000 UTC m=+645.189656122" watchObservedRunningTime="2026-11-03 18:49:30.309160158 +0000 UTC m=+645.190576317" Nov 03 18:49:31.284168 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:31.284136 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 18:49:31.284547 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:31.284196 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 18:49:32.284423 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:32.284391 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 18:49:32.284848 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:32.284443 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 18:49:33.284833 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:33.284797 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 18:49:33.285212 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:33.284853 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 18:49:34.284282 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:34.284243 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 18:49:34.284569 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:34.284300 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 18:49:35.284882 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:35.284850 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 18:49:35.285275 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:35.284902 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 18:49:36.284252 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:36.284216 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 18:49:36.284427 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:36.284270 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 18:49:38.285370 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:38.285334 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:49:38.285836 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:38.285423 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:49:38.385567 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:38.385534 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:49:39.326832 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:39.326800 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6t8sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:49:39.327300 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:39.326848 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:49:39.783624 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:39.783535 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:39.783972 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:39.783949 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:39.784771 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:39.784753 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:49:43.956575 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:43.956511 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:55.429769 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:55.429729 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:55.646699 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:49:55.646642 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:49:55.943807 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:49:55.943768 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:50:06.484845 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:06.484812 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:06.724018 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:06.723981 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:25.937155 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:50:25.937111 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:50:30.137801 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:30.137769 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:30.346593 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:30.346558 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:31.884764 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:31.884731 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:32.101626 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:32.101595 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:33.509079 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:33.509044 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:33.783294 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:33.783211 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:44.274869 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:44.274827 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:44.481198 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:44.481164 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:55.935936 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:50:55.935854 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:50:58.169495 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:58.169454 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:58.383192 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:58.383162 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:58.739126 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:58.739082 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:58.969545 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:58.969509 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:59.209230 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:59.209189 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:59.413968 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:59.413934 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:59.665870 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:59.665835 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:50:59.881650 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:50:59.881614 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:00.091008 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:00.090969 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:00.299060 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:00.299026 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:00.503510 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:00.503415 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:00.712234 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:00.712200 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:00.917093 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:00.917038 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:03.621934 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:03.621899 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:03.828262 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:03.828226 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:04.336587 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:04.336556 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:04.565650 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:04.565621 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:05.051783 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:05.051751 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:05.257898 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:05.257864 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:05.463723 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:05.463672 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:05.672041 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:05.672007 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:05.887700 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:05.887659 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:06.326807 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:06.326770 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:06.528857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:06.528821 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:06.878698 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:06.878653 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:07.090437 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:07.090402 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:09.112798 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.112764 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:09.310522 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.310490 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:09.520666 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.520589 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z"] Nov 03 18:51:09.524182 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.524152 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.527711 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.527655 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:09.573191 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.573150 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-2e052a-serving-cert\"" Nov 03 18:51:09.619624 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.619600 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:51:09.619795 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.619659 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\"" Nov 03 18:51:09.620848 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.620828 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z"] Nov 03 18:51:09.632557 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.632525 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.632711 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.632575 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da384f09-39e5-4c50-872c-6cc981339aa0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.632773 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.632737 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-metrics-certs\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.632811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.632782 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4v5f\" (UniqueName: \"kubernetes.io/projected/da384f09-39e5-4c50-872c-6cc981339aa0-kube-api-access-q4v5f\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.733771 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.733738 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.733918 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.733785 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da384f09-39e5-4c50-872c-6cc981339aa0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.733918 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.733832 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-metrics-certs\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.733918 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.733860 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4v5f\" (UniqueName: \"kubernetes.io/projected/da384f09-39e5-4c50-872c-6cc981339aa0-kube-api-access-q4v5f\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.733918 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:09.733897 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-serving-cert: secret "maas-api-e2e-worker-w4-2e052a-serving-cert" not found Nov 03 18:51:09.734049 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:09.733979 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls podName:da384f09-39e5-4c50-872c-6cc981339aa0 nodeName:}" failed. No retries permitted until 2026-11-03 18:51:10.233962705 +0000 UTC m=+745.115378785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls") pod "maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" (UID: "da384f09-39e5-4c50-872c-6cc981339aa0") : secret "maas-api-e2e-worker-w4-2e052a-serving-cert" not found Nov 03 18:51:09.734532 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.734515 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da384f09-39e5-4c50-872c-6cc981339aa0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.736181 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.736164 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-metrics-certs\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.785390 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.785316 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:09.790828 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.790808 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4v5f\" (UniqueName: \"kubernetes.io/projected/da384f09-39e5-4c50-872c-6cc981339aa0-kube-api-access-q4v5f\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:09.995343 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:09.995302 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:10.209613 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:10.209580 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:10.239866 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:10.239838 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:10.242317 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:10.242285 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:10.441293 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:10.441261 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:10.806416 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:10.806389 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z"] Nov 03 18:51:10.809165 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:51:10.809143 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda384f09_39e5_4c50_872c_6cc981339aa0.slice/crio-7cfe5e389006a8a11a760c2a5365b491b6a774f0971dd79624a1279b1be11f68 WatchSource:0}: Error finding container 7cfe5e389006a8a11a760c2a5365b491b6a774f0971dd79624a1279b1be11f68: Status 404 returned error can't find the container with id 7cfe5e389006a8a11a760c2a5365b491b6a774f0971dd79624a1279b1be11f68 Nov 03 18:51:10.810486 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:10.810467 5988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:51:11.637237 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:11.637201 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" event={"ID":"da384f09-39e5-4c50-872c-6cc981339aa0","Type":"ContainerStarted","Data":"7cfe5e389006a8a11a760c2a5365b491b6a774f0971dd79624a1279b1be11f68"} Nov 03 18:51:12.643491 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:12.643450 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" event={"ID":"da384f09-39e5-4c50-872c-6cc981339aa0","Type":"ContainerStarted","Data":"7b5e0230458f463affa347f4d8af293134e73168fe9f8fd99602b18dc1b2276f"} Nov 03 18:51:12.643889 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:12.643521 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:12.754972 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:12.754927 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" podStartSLOduration=2.833454538 podStartE2EDuration="3.754913617s" totalImagesPullingTime="921.459079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:12.754506368 +0000 UTC m=+747.635922474" watchObservedRunningTime="2026-11-03 18:51:12.754913617 +0000 UTC m=+747.636329720" Nov 03 18:51:14.258256 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:14.258227 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:14.578498 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:14.578471 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:14.919106 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:14.919027 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:15.138873 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:15.138835 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:16.964050 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:16.964012 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:17.191485 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:17.191447 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:18.024603 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:18.024565 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:51:18.653912 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:18.653884 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 18:51:19.520294 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:19.520260 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:19.730618 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:19.730587 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:19.945555 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:19.945520 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:20.011534 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.011499 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4"] Nov 03 18:51:20.015636 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.015615 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.064708 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.064654 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-355e1a\"" Nov 03 18:51:20.111003 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.110975 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-355e1a-dockercfg-b8ztx\"" Nov 03 18:51:20.112315 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.112290 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4"] Nov 03 18:51:20.133534 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.133496 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa7aae1f-b048-4e60-94b4-0831a6e24c92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.133715 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.133545 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxwm\" (UniqueName: \"kubernetes.io/projected/fa7aae1f-b048-4e60-94b4-0831a6e24c92-kube-api-access-9zxwm\") pod \"payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.148636 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.148605 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:20.234591 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.234503 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxwm\" (UniqueName: \"kubernetes.io/projected/fa7aae1f-b048-4e60-94b4-0831a6e24c92-kube-api-access-9zxwm\") pod \"payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.234778 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.234615 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa7aae1f-b048-4e60-94b4-0831a6e24c92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.235349 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.235328 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa7aae1f-b048-4e60-94b4-0831a6e24c92-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.292359 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.292324 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxwm\" (UniqueName: \"kubernetes.io/projected/fa7aae1f-b048-4e60-94b4-0831a6e24c92-kube-api-access-9zxwm\") pod \"payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.330038 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.330004 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:20.693305 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:20.693277 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4"] Nov 03 18:51:20.696778 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:51:20.696744 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7aae1f_b048_4e60_94b4_0831a6e24c92.slice/crio-e09ade9e11b1b355082b29673ba8ba36c8592d7a4e542de26b60355ee3d1a3e3 WatchSource:0}: Error finding container e09ade9e11b1b355082b29673ba8ba36c8592d7a4e542de26b60355ee3d1a3e3: Status 404 returned error can't find the container with id e09ade9e11b1b355082b29673ba8ba36c8592d7a4e542de26b60355ee3d1a3e3 Nov 03 18:51:21.688010 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:21.687978 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" event={"ID":"fa7aae1f-b048-4e60-94b4-0831a6e24c92","Type":"ContainerStarted","Data":"e09ade9e11b1b355082b29673ba8ba36c8592d7a4e542de26b60355ee3d1a3e3"} Nov 03 18:51:22.179909 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.179864 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:22.403770 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.403735 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:22.701490 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.701457 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv"] Nov 03 18:51:22.711024 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.710993 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:22.760936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.760897 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv"] Nov 03 18:51:22.805706 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.805643 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-32a59b\"" Nov 03 18:51:22.805899 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.805770 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-32a59b-dockercfg-mzvkk\"" Nov 03 18:51:22.872135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.872093 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf096dc-a935-4576-8fbf-74326af2d009-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:22.872335 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.872152 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87jr\" (UniqueName: \"kubernetes.io/projected/3cf096dc-a935-4576-8fbf-74326af2d009-kube-api-access-j87jr\") pod \"payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:22.972984 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.972896 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf096dc-a935-4576-8fbf-74326af2d009-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:22.972984 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.972948 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87jr\" (UniqueName: \"kubernetes.io/projected/3cf096dc-a935-4576-8fbf-74326af2d009-kube-api-access-j87jr\") pod \"payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:22.973694 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:22.973661 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf096dc-a935-4576-8fbf-74326af2d009-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:23.030905 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:23.030866 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87jr\" (UniqueName: \"kubernetes.io/projected/3cf096dc-a935-4576-8fbf-74326af2d009-kube-api-access-j87jr\") pod \"payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:23.329275 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:23.329235 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:23.694321 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:23.694251 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv"] Nov 03 18:51:23.697511 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:51:23.697482 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf096dc_a935_4576_8fbf_74326af2d009.slice/crio-1a14dd85650e7d992a6b863e49120015a206ab6ac20b157fb3ae5e20a10c8d6a WatchSource:0}: Error finding container 1a14dd85650e7d992a6b863e49120015a206ab6ac20b157fb3ae5e20a10c8d6a: Status 404 returned error can't find the container with id 1a14dd85650e7d992a6b863e49120015a206ab6ac20b157fb3ae5e20a10c8d6a Nov 03 18:51:23.701569 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:23.701539 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" event={"ID":"fa7aae1f-b048-4e60-94b4-0831a6e24c92","Type":"ContainerStarted","Data":"611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa"} Nov 03 18:51:23.701668 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:23.701629 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:23.748854 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:23.748830 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:51:23.813908 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:23.813103 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" podStartSLOduration=2.103957323 podStartE2EDuration="4.813067358s" totalImagesPullingTime="2.709110035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:23.812567137 +0000 UTC m=+758.693983238" watchObservedRunningTime="2026-11-03 18:51:23.813067358 +0000 UTC m=+758.694483471" Nov 03 18:51:24.434174 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.434141 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k"] Nov 03 18:51:24.437936 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.437915 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.487109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.487079 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-32a59b-serving-cert\"" Nov 03 18:51:24.534698 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.534651 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k"] Nov 03 18:51:24.587036 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.587005 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.587189 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.587123 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-metrics-certs\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.587238 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.587198 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f118efc3-0ece-4311-a4be-6432b720e19a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.587238 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.587234 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgr25\" (UniqueName: \"kubernetes.io/projected/f118efc3-0ece-4311-a4be-6432b720e19a-kube-api-access-wgr25\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.687602 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.687522 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:24.687781 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.687615 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f118efc3-0ece-4311-a4be-6432b720e19a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.687781 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.687664 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgr25\" (UniqueName: \"kubernetes.io/projected/f118efc3-0ece-4311-a4be-6432b720e19a-kube-api-access-wgr25\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.687781 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.687730 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.687953 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.687803 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-metrics-certs\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.687953 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:24.687905 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-serving-cert: secret "maas-api-e2e-worker-w2-32a59b-serving-cert" not found Nov 03 18:51:24.688073 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:24.687956 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls podName:f118efc3-0ece-4311-a4be-6432b720e19a nodeName:}" failed. No retries permitted until 2026-11-03 18:51:25.187943974 +0000 UTC m=+760.069360053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls") pod "maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" (UID: "f118efc3-0ece-4311-a4be-6432b720e19a") : secret "maas-api-e2e-worker-w2-32a59b-serving-cert" not found Nov 03 18:51:24.688257 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.688237 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f118efc3-0ece-4311-a4be-6432b720e19a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.690105 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.690082 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-metrics-certs\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.707376 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.707349 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" event={"ID":"3cf096dc-a935-4576-8fbf-74326af2d009","Type":"ContainerStarted","Data":"5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac"} Nov 03 18:51:24.707510 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.707494 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:24.707561 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.707530 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" event={"ID":"3cf096dc-a935-4576-8fbf-74326af2d009","Type":"ContainerStarted","Data":"1a14dd85650e7d992a6b863e49120015a206ab6ac20b157fb3ae5e20a10c8d6a"} Nov 03 18:51:24.745341 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.745325 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgr25\" (UniqueName: \"kubernetes.io/projected/f118efc3-0ece-4311-a4be-6432b720e19a-kube-api-access-wgr25\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:24.817590 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.817539 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" podStartSLOduration=2.817523439 podStartE2EDuration="2.817523439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:24.816050178 +0000 UTC m=+759.697466281" watchObservedRunningTime="2026-11-03 18:51:24.817523439 +0000 UTC m=+759.698939539" Nov 03 18:51:24.897103 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:24.897066 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:25.102344 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:25.102306 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:25.191531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:25.191496 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:25.194060 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:25.194034 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:25.309913 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:25.309878 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:25.351453 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:25.351425 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:25.766772 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:25.766707 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k"] Nov 03 18:51:25.769500 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:51:25.769472 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf118efc3_0ece_4311_a4be_6432b720e19a.slice/crio-1ab05c82a518d1a4e4233729483469bafee240dcf18938e2a7df644563580e47 WatchSource:0}: Error finding container 1ab05c82a518d1a4e4233729483469bafee240dcf18938e2a7df644563580e47: Status 404 returned error can't find the container with id 1ab05c82a518d1a4e4233729483469bafee240dcf18938e2a7df644563580e47 Nov 03 18:51:25.936108 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:25.936072 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:51:26.718281 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:26.718246 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" event={"ID":"f118efc3-0ece-4311-a4be-6432b720e19a","Type":"ContainerStarted","Data":"3895ef973bf0480c8436004240761e2fd68ebec8dcbaae9ec9b098220f98a05c"} Nov 03 18:51:26.718448 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:26.718301 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" event={"ID":"f118efc3-0ece-4311-a4be-6432b720e19a","Type":"ContainerStarted","Data":"1ab05c82a518d1a4e4233729483469bafee240dcf18938e2a7df644563580e47"} Nov 03 18:51:26.718448 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:26.718362 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:26.841757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:26.841640 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" podStartSLOduration=2.037406462 podStartE2EDuration="2.841624461s" totalImagesPullingTime="804.217999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:26.840518992 +0000 UTC m=+761.721935097" watchObservedRunningTime="2026-11-03 18:51:26.841624461 +0000 UTC m=+761.723040562" Nov 03 18:51:27.343604 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:27.343571 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:27.544144 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:27.544116 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:29.515289 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:29.515252 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:32.507443 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:32.507410 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:32.706679 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:32.706650 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:32.728529 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:32.728505 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 18:51:34.383984 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:34.383952 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:34.597659 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:34.597610 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:34.709983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:34.709920 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 18:51:34.836775 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:34.836741 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:35.715368 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:35.715343 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 18:51:38.814960 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.814925 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc"] Nov 03 18:51:38.821767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.821744 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:38.870727 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.870678 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-fba266-serving-cert\"" Nov 03 18:51:38.918305 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.918273 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc"] Nov 03 18:51:38.919884 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.919865 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-metrics-certs\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:38.919967 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.919895 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f119ec89-accf-40b3-87f7-dcf59d978d0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:38.919967 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.919929 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:38.920124 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:38.920100 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq82d\" (UniqueName: \"kubernetes.io/projected/f119ec89-accf-40b3-87f7-dcf59d978d0f-kube-api-access-pq82d\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.021070 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.021035 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.021257 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.021139 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq82d\" (UniqueName: \"kubernetes.io/projected/f119ec89-accf-40b3-87f7-dcf59d978d0f-kube-api-access-pq82d\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.021257 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:39.021179 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-serving-cert: secret "maas-api-e2e-worker-w6-fba266-serving-cert" not found Nov 03 18:51:39.021257 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.021204 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-metrics-certs\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.021257 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:39.021242 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls podName:f119ec89-accf-40b3-87f7-dcf59d978d0f nodeName:}" failed. No retries permitted until 2026-11-03 18:51:39.521221129 +0000 UTC m=+774.402637216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls") pod "maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" (UID: "f119ec89-accf-40b3-87f7-dcf59d978d0f") : secret "maas-api-e2e-worker-w6-fba266-serving-cert" not found Nov 03 18:51:39.021508 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.021260 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f119ec89-accf-40b3-87f7-dcf59d978d0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.021983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.021959 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f119ec89-accf-40b3-87f7-dcf59d978d0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.025434 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.025412 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-metrics-certs\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.065704 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.065622 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:39.080098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.080076 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq82d\" (UniqueName: \"kubernetes.io/projected/f119ec89-accf-40b3-87f7-dcf59d978d0f-kube-api-access-pq82d\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.280244 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.280206 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:39.525041 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.525004 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.527331 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.527312 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:39.735487 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:39.735448 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:40.102587 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:40.102458 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc"] Nov 03 18:51:40.105228 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:51:40.105200 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf119ec89_accf_40b3_87f7_dcf59d978d0f.slice/crio-755849a6d3ba1d83fce41801bb27005c0606050fbdd7f543040d9eb9ecc2e0c0 WatchSource:0}: Error finding container 755849a6d3ba1d83fce41801bb27005c0606050fbdd7f543040d9eb9ecc2e0c0: Status 404 returned error can't find the container with id 755849a6d3ba1d83fce41801bb27005c0606050fbdd7f543040d9eb9ecc2e0c0 Nov 03 18:51:40.783995 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:40.783957 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" event={"ID":"f119ec89-accf-40b3-87f7-dcf59d978d0f","Type":"ContainerStarted","Data":"755849a6d3ba1d83fce41801bb27005c0606050fbdd7f543040d9eb9ecc2e0c0"} Nov 03 18:51:41.790211 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:41.790177 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" event={"ID":"f119ec89-accf-40b3-87f7-dcf59d978d0f","Type":"ContainerStarted","Data":"4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864"} Nov 03 18:51:41.790607 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:41.790240 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:41.901983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:41.901933 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" podStartSLOduration=3.062187873 podStartE2EDuration="3.901916602s" totalImagesPullingTime="839.728729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:51:41.90067132 +0000 UTC m=+776.782087425" watchObservedRunningTime="2026-11-03 18:51:41.901916602 +0000 UTC m=+776.783332704" Nov 03 18:51:43.836730 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:43.836697 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:51:43.851528 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:43.851499 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:44.702767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:44.702731 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:47.800946 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:47.800917 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 18:51:48.757979 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:48.757947 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:53.611679 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:53.611641 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:51:53.732799 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:53.732758 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:53.943293 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:53.943214 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:54.157544 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:54.157510 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:54.397547 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:54.397518 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:54.727993 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:54.727914 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:54.963374 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:54.963347 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:55.169751 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:55.169711 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:55.935142 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:51:55.935106 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:51:58.617546 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.617506 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h"] Nov 03 18:51:58.621473 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.621452 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.670945 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.670920 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"openshift-service-ca.crt\"" Nov 03 18:51:58.699366 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.699335 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-dshm\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.699512 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.699415 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tls-certs\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.699512 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.699491 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tmp-dir\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.699592 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.699553 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgcb\" (UniqueName: \"kubernetes.io/projected/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-kube-api-access-4wgcb\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.699626 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.699594 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-home\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.699675 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.699661 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-model-cache\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.717224 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.717202 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"kube-root-ca.crt\"" Nov 03 18:51:58.717384 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.717370 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"default-dockercfg-dbh7c\"" Nov 03 18:51:58.717472 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.717449 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2e052a\"/\"simulator-w4-2e052a-kserve-self-signed-certs\"" Nov 03 18:51:58.718868 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.718851 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h"] Nov 03 18:51:58.763665 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.763635 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:58.800998 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.800968 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-dshm\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801108 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801068 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tls-certs\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801149 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801105 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tmp-dir\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801199 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801183 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgcb\" (UniqueName: \"kubernetes.io/projected/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-kube-api-access-4wgcb\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801238 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801211 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-home\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801289 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801266 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-model-cache\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801554 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801522 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tmp-dir\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801672 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801554 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-home\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.801753 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.801660 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-model-cache\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.803259 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.803240 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-dshm\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.803531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.803514 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tls-certs\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.858646 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.858625 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgcb\" (UniqueName: \"kubernetes.io/projected/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-kube-api-access-4wgcb\") pod \"simulator-w4-2e052a-kserve-7c4c9857f4-rp27h\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:58.934747 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:58.932744 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:51:59.096605 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:59.096550 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:51:59.309305 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:59.309282 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h"] Nov 03 18:51:59.311900 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:51:59.311875 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ed9f5e1_ead3_4a19_9ed1_25687b8db664.slice/crio-772ab499c4cafd97762edaa5938c56db75e0068e274faf9e51d07c13db9eed2b WatchSource:0}: Error finding container 772ab499c4cafd97762edaa5938c56db75e0068e274faf9e51d07c13db9eed2b: Status 404 returned error can't find the container with id 772ab499c4cafd97762edaa5938c56db75e0068e274faf9e51d07c13db9eed2b Nov 03 18:51:59.877670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:51:59.877637 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" event={"ID":"6ed9f5e1-ead3-4a19-9ed1-25687b8db664","Type":"ContainerStarted","Data":"772ab499c4cafd97762edaa5938c56db75e0068e274faf9e51d07c13db9eed2b"} Nov 03 18:52:01.891085 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:01.891048 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" event={"ID":"6ed9f5e1-ead3-4a19-9ed1-25687b8db664","Type":"ContainerStarted","Data":"7ddd0d3e47724a9077721df62ddbb26c709a2a6435fdf837d8f9cc71b798f5ff"} Nov 03 18:52:01.933408 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:01.933371 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:52:02.124673 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:02.124614 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:03.819313 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:03.819277 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:04.928529 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:04.928493 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:05.535975 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:05.535938 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:05.756416 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:05.756378 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:08.862564 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:08.862528 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:08.934134 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:08.934097 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:52:08.946880 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:08.946852 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:52:09.057795 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:09.057740 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podStartSLOduration=9.296070003 podStartE2EDuration="11.057725039s" totalImagesPullingTime="1.761655036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:02.004400544 +0000 UTC m=+796.885816646" watchObservedRunningTime="2026-11-03 18:52:09.057725039 +0000 UTC m=+803.939141140" Nov 03 18:52:09.942852 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:09.942822 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 18:52:10.374428 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:10.374394 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:10.943640 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:10.943609 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:11.149709 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:11.149649 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:13.952009 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:13.951975 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:14.562894 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:14.562860 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:52:18.623720 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:18.623667 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:20.809147 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:20.809096 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:21.386969 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:21.386928 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:21.665713 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:21.665618 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:23.770274 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:23.770231 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:24.976378 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:24.976340 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:25.935691 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:52:25.935646 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:52:27.766841 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:27.766800 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" podUID="e644f9f1-4403-4823-a83a-e1822f286a97" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.34:8081/healthz\": read tcp 10.134.0.2:52688->10.134.0.34:8081: read: connection reset by peer" Nov 03 18:52:28.016660 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:28.016631 5988 generic.go:356] "Generic (PLEG): container finished" podID="e644f9f1-4403-4823-a83a-e1822f286a97" containerID="b1f02cc801a0bb5dcf2c9e7b86afdd2b681571b776f9fe60434b9d6ba7649a6e" exitCode=1 Nov 03 18:52:28.016886 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:28.016710 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" event={"ID":"e644f9f1-4403-4823-a83a-e1822f286a97","Type":"ContainerDied","Data":"b1f02cc801a0bb5dcf2c9e7b86afdd2b681571b776f9fe60434b9d6ba7649a6e"} Nov 03 18:52:28.017074 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:28.017058 5988 scope.go:122] "RemoveContainer" containerID="b1f02cc801a0bb5dcf2c9e7b86afdd2b681571b776f9fe60434b9d6ba7649a6e" Nov 03 18:52:28.499363 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:28.499327 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:29.023476 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:29.023439 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" event={"ID":"e644f9f1-4403-4823-a83a-e1822f286a97","Type":"ContainerStarted","Data":"60ca77b3923e9f9f63d77c11d5d08ba563d0965ead7ffb9791302e2159e2d981"} Nov 03 18:52:29.023898 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:29.023672 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:52:33.000072 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:33.000040 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:35.558325 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:35.558287 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:35.771298 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:35.771266 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:37.602907 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:37.602877 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:41.783637 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:41.783599 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:42.395087 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:42.395055 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:42.603415 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:42.603381 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:45.060907 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:45.060877 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:47.077325 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:47.077291 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:51.814916 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:51.814857 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:52:55.936813 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:52:55.936774 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:52:56.433057 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:52:56.433025 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:00.030479 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:00.030444 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7z57r" Nov 03 18:53:01.111852 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:01.111821 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:05.616956 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:05.616918 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:10.227102 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:10.227052 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:14.800026 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:14.799960 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:19.392438 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:19.392394 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:22.896363 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:22.896329 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2"] Nov 03 18:53:22.900277 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:22.900255 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:22.952350 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:22.952326 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"default-dockercfg-h7sxm\"" Nov 03 18:53:22.994066 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:22.994044 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"kube-root-ca.crt\"" Nov 03 18:53:22.994189 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:22.994095 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"openshift-service-ca.crt\"" Nov 03 18:53:23.000526 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.000497 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2"] Nov 03 18:53:23.011557 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.011539 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-model-cache\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.011729 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.011565 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cg5x\" (UniqueName: \"kubernetes.io/projected/23929892-bb70-4d89-9d34-fbeb52b9759f-kube-api-access-2cg5x\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.011729 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.011588 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-home\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.011729 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.011610 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-dshm\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.011876 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.011772 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-tmp-dir\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.011876 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.011837 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23929892-bb70-4d89-9d34-fbeb52b9759f-tls-certs\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.048714 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.048665 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:23.050504 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.050489 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-3a7056\"/\"simulator-w5-3a7056-kserve-self-signed-certs\"" Nov 03 18:53:23.112601 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.112572 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23929892-bb70-4d89-9d34-fbeb52b9759f-tls-certs\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.112786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.112621 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-model-cache\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.112786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.112649 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cg5x\" (UniqueName: \"kubernetes.io/projected/23929892-bb70-4d89-9d34-fbeb52b9759f-kube-api-access-2cg5x\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.112786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.112714 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-home\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.112786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.112749 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-dshm\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.113016 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.112850 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-tmp-dir\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.113098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.113076 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-model-cache\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.113172 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.113107 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-home\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.113288 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.113265 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-tmp-dir\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.115227 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.115199 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-dshm\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.115323 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.115241 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23929892-bb70-4d89-9d34-fbeb52b9759f-tls-certs\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.170380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.170320 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cg5x\" (UniqueName: \"kubernetes.io/projected/23929892-bb70-4d89-9d34-fbeb52b9759f-kube-api-access-2cg5x\") pod \"simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.212111 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.212085 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:23.637571 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:23.637538 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2"] Nov 03 18:53:23.675747 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:53:23.675722 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23929892_bb70_4d89_9d34_fbeb52b9759f.slice/crio-2d8e4618b6daa01987f1e496cf087ed73ab3cf04826db2ce67728e363660bb67 WatchSource:0}: Error finding container 2d8e4618b6daa01987f1e496cf087ed73ab3cf04826db2ce67728e363660bb67: Status 404 returned error can't find the container with id 2d8e4618b6daa01987f1e496cf087ed73ab3cf04826db2ce67728e363660bb67 Nov 03 18:53:24.050343 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:24.050257 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:24.306610 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:24.306532 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" event={"ID":"23929892-bb70-4d89-9d34-fbeb52b9759f","Type":"ContainerStarted","Data":"95e957d09fc563019f49ee7a570e37febcafa9d829438702dde2ef8666cdcc62"} Nov 03 18:53:24.306610 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:24.306573 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" event={"ID":"23929892-bb70-4d89-9d34-fbeb52b9759f","Type":"ContainerStarted","Data":"2d8e4618b6daa01987f1e496cf087ed73ab3cf04826db2ce67728e363660bb67"} Nov 03 18:53:24.417109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:24.417057 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podStartSLOduration=2.417043962 podStartE2EDuration="2.417043962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:24.416046293 +0000 UTC m=+879.297462406" watchObservedRunningTime="2026-11-03 18:53:24.417043962 +0000 UTC m=+879.298460075" Nov 03 18:53:25.212601 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:25.212572 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:25.935910 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:53:25.935875 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:53:26.069461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:26.069429 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:28.697950 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:28.697913 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:30.731420 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:30.731388 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:31.085829 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:31.085794 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:53:33.212234 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:33.212197 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:33.225056 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:33.225025 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:33.300749 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:33.300712 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:33.362510 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:33.362482 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 18:53:37.821548 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:37.821510 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:39.755397 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.755357 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn"] Nov 03 18:53:39.759409 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.759388 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.809701 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.809660 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"openshift-service-ca.crt\"" Nov 03 18:53:39.809842 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.809707 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"kube-root-ca.crt\"" Nov 03 18:53:39.856293 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.856254 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"default-dockercfg-qbv72\"" Nov 03 18:53:39.856499 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.856472 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"simulator-w6-fba266-kserve-self-signed-certs\"" Nov 03 18:53:39.859927 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.859896 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn"] Nov 03 18:53:39.873777 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.873752 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-home\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.873879 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.873828 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgvh\" (UniqueName: \"kubernetes.io/projected/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-kube-api-access-mxgvh\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.873926 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.873904 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-dshm\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.873961 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.873947 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tmp-dir\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.874010 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.873976 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tls-certs\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.874088 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.874069 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-model-cache\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.914626 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.914589 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:39.975008 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.974980 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgvh\" (UniqueName: \"kubernetes.io/projected/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-kube-api-access-mxgvh\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975184 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975022 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-dshm\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975184 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975123 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tmp-dir\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975184 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975146 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tls-certs\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975184 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975173 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-model-cache\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975415 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975211 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-home\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975615 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975591 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-model-cache\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975768 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975618 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tmp-dir\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.975768 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.975667 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-home\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.977265 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.977243 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-dshm\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:39.977511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:39.977493 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tls-certs\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:40.033248 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:40.033163 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgvh\" (UniqueName: \"kubernetes.io/projected/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-kube-api-access-mxgvh\") pod \"simulator-w6-fba266-kserve-84764bc7cb-ww7rn\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:40.073037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:40.073000 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:40.449806 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:40.449769 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn"] Nov 03 18:53:40.500130 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:53:40.500098 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod295e4c83_5d79_4f4a_9e3f_fac2c816f4b7.slice/crio-7e5ca12bc3520cb52280477143ee7fb3d5aadbea99267e62b73a6c4d90f63538 WatchSource:0}: Error finding container 7e5ca12bc3520cb52280477143ee7fb3d5aadbea99267e62b73a6c4d90f63538: Status 404 returned error can't find the container with id 7e5ca12bc3520cb52280477143ee7fb3d5aadbea99267e62b73a6c4d90f63538 Nov 03 18:53:41.393842 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:41.393808 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" event={"ID":"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7","Type":"ContainerStarted","Data":"55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe"} Nov 03 18:53:41.394234 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:41.393851 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" event={"ID":"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7","Type":"ContainerStarted","Data":"7e5ca12bc3520cb52280477143ee7fb3d5aadbea99267e62b73a6c4d90f63538"} Nov 03 18:53:41.507195 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:41.507135 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podStartSLOduration=2.507121665 podStartE2EDuration="2.507121665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:41.504755405 +0000 UTC m=+896.386171508" watchObservedRunningTime="2026-11-03 18:53:41.507121665 +0000 UTC m=+896.388537766" Nov 03 18:53:42.073650 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:42.073618 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:42.494057 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:42.493980 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:45.140406 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:45.140372 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:45.424111 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:45.424035 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:47.110308 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:47.110272 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:50.074245 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:50.074210 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:50.086491 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:50.086467 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:50.450421 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:50.450347 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 18:53:50.753970 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:50.753878 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:51.706975 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:51.706940 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:55.937169 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:53:55.937087 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:53:56.409670 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:56.409638 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:53:58.427972 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:53:58.427938 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:54:01.011555 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:01.011518 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:05.078293 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.078263 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw"] Nov 03 18:54:05.082090 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.082071 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.138557 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.138481 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"kube-root-ca.crt\"" Nov 03 18:54:05.177761 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.177728 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"openshift-service-ca.crt\"" Nov 03 18:54:05.177761 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.177732 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"default-dockercfg-vzdn4\"" Nov 03 18:54:05.177761 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.177755 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a4cacd\"/\"premium-w3-a4cacd-kserve-self-signed-certs\"" Nov 03 18:54:05.179328 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.179310 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw"] Nov 03 18:54:05.224577 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.224551 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856af342-3dc8-4538-a6a0-e779029486a7-tls-certs\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.224731 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.224597 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-tmp-dir\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.224731 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.224615 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-model-cache\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.224817 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.224772 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xhj\" (UniqueName: \"kubernetes.io/projected/856af342-3dc8-4538-a6a0-e779029486a7-kube-api-access-99xhj\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.224857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.224835 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-home\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.224957 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.224939 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-dshm\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.231780 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.231749 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:05.326195 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326170 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-dshm\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326353 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326217 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856af342-3dc8-4538-a6a0-e779029486a7-tls-certs\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326353 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326254 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-tmp-dir\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326353 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326274 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-model-cache\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326353 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326303 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xhj\" (UniqueName: \"kubernetes.io/projected/856af342-3dc8-4538-a6a0-e779029486a7-kube-api-access-99xhj\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326353 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326343 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-home\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326727 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326701 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-tmp-dir\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326842 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326728 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-model-cache\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.326842 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.326784 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-home\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.328775 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.328716 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-dshm\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.328934 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.328918 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856af342-3dc8-4538-a6a0-e779029486a7-tls-certs\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.384205 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.384178 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xhj\" (UniqueName: \"kubernetes.io/projected/856af342-3dc8-4538-a6a0-e779029486a7-kube-api-access-99xhj\") pod \"premium-w3-a4cacd-kserve-bc459cfd9-zkkkw\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.394053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.394024 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:05.569807 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.569779 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:05.773218 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.773135 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw"] Nov 03 18:54:05.780677 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:05.780644 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:05.807631 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:54:05.807609 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod856af342_3dc8_4538_a6a0_e779029486a7.slice/crio-351b9a9ba107d9b927aa179798598ad141765d98fcb34c9691162d1678b5a20a WatchSource:0}: Error finding container 351b9a9ba107d9b927aa179798598ad141765d98fcb34c9691162d1678b5a20a: Status 404 returned error can't find the container with id 351b9a9ba107d9b927aa179798598ad141765d98fcb34c9691162d1678b5a20a Nov 03 18:54:06.529506 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:06.529444 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" event={"ID":"856af342-3dc8-4538-a6a0-e779029486a7","Type":"ContainerStarted","Data":"34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd"} Nov 03 18:54:06.529506 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:06.529507 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" event={"ID":"856af342-3dc8-4538-a6a0-e779029486a7","Type":"ContainerStarted","Data":"351b9a9ba107d9b927aa179798598ad141765d98fcb34c9691162d1678b5a20a"} Nov 03 18:54:06.641211 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:06.641162 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podStartSLOduration=1.6411489540000002 podStartE2EDuration="1.641148954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:06.639746713 +0000 UTC m=+921.521162816" watchObservedRunningTime="2026-11-03 18:54:06.641148954 +0000 UTC m=+921.522565055" Nov 03 18:54:07.394582 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:07.394552 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:10.288367 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:10.288336 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:14.905399 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:14.905368 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:15.394877 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:15.394846 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:15.407258 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:15.407214 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:15.590966 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:15.590939 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 18:54:19.510988 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:19.510955 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:24.128318 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:24.128281 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:54:24.140864 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:24.140806 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:25.935824 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:54:25.935777 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:54:28.733957 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:28.733920 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:33.321102 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:33.321059 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:34.120561 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:34.120526 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:37.910892 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:37.910847 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:42.524669 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:42.524629 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:45.819092 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:45.819059 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:47.144042 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:47.144008 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:49.496279 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:49.496246 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:49.705930 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:49.705860 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:51.701119 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:51.701036 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:52.169712 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:52.169647 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:54:55.619036 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.618992 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8"] Nov 03 18:54:55.623462 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.623433 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.672841 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.672809 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"default-dockercfg-zr88h\"" Nov 03 18:54:55.718502 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.718472 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"openshift-service-ca.crt\"" Nov 03 18:54:55.719281 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.719261 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"premium-w1-355e1a-kserve-self-signed-certs\"" Nov 03 18:54:55.719364 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.719317 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"kube-root-ca.crt\"" Nov 03 18:54:55.720860 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.720840 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8"] Nov 03 18:54:55.727915 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.727889 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-model-cache\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.728005 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.727934 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-home\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.728054 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.728003 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnww\" (UniqueName: \"kubernetes.io/projected/65d04039-cc62-4e81-b1e6-8d0d58b41704-kube-api-access-hvnww\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.728113 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.728087 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d04039-cc62-4e81-b1e6-8d0d58b41704-tls-certs\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.728177 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.728159 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-dshm\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.728224 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.728211 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-tmp-dir\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.767516 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.767469 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:55.829076 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829041 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnww\" (UniqueName: \"kubernetes.io/projected/65d04039-cc62-4e81-b1e6-8d0d58b41704-kube-api-access-hvnww\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829101 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d04039-cc62-4e81-b1e6-8d0d58b41704-tls-certs\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829141 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-dshm\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829180 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-tmp-dir\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829221 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-model-cache\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829264 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-home\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829829 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829802 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-home\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829847 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-model-cache\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.829943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.829842 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-tmp-dir\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.831437 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.831417 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-dshm\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.832114 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.832098 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d04039-cc62-4e81-b1e6-8d0d58b41704-tls-certs\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.888480 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.888387 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnww\" (UniqueName: \"kubernetes.io/projected/65d04039-cc62-4e81-b1e6-8d0d58b41704-kube-api-access-hvnww\") pod \"premium-w1-355e1a-kserve-7c74898bb5-vbwk8\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.936708 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:55.936442 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:54:55.939281 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:54:55.939230 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:54:56.320533 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:56.320497 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8"] Nov 03 18:54:56.366525 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:56.366398 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:54:56.371319 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:54:56.371290 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65d04039_cc62_4e81_b1e6_8d0d58b41704.slice/crio-3eb67c2c02e00bd18ef789c88782d242a59dbc97ae2891cf06f20f383fe9bc85 WatchSource:0}: Error finding container 3eb67c2c02e00bd18ef789c88782d242a59dbc97ae2891cf06f20f383fe9bc85: Status 404 returned error can't find the container with id 3eb67c2c02e00bd18ef789c88782d242a59dbc97ae2891cf06f20f383fe9bc85 Nov 03 18:54:56.801929 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:56.801831 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" event={"ID":"65d04039-cc62-4e81-b1e6-8d0d58b41704","Type":"ContainerStarted","Data":"4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c"} Nov 03 18:54:56.801929 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:56.801875 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" event={"ID":"65d04039-cc62-4e81-b1e6-8d0d58b41704","Type":"ContainerStarted","Data":"3eb67c2c02e00bd18ef789c88782d242a59dbc97ae2891cf06f20f383fe9bc85"} Nov 03 18:54:56.914437 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:56.914375 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podStartSLOduration=1.914358821 podStartE2EDuration="1.914358821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:56.913485974 +0000 UTC m=+971.794902076" watchObservedRunningTime="2026-11-03 18:54:56.914358821 +0000 UTC m=+971.795774941" Nov 03 18:54:56.937192 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:54:56.937156 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:55:01.026510 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:01.026448 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:05.052491 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:05.052456 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:05.641452 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:05.641388 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:05.937079 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:05.936991 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:55:05.949290 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:05.949260 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:55:06.866961 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:06.866930 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 18:55:08.579173 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:08.579127 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:08.839593 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:08.839470 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:10.305531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:10.305461 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:11.831469 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:11.831431 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:14.916380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:14.916348 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:17.104312 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.104278 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl"] Nov 03 18:55:17.107816 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.107791 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.179917 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.179888 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"premium-w0-9686c3-kserve-self-signed-certs\"" Nov 03 18:55:17.179917 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.179917 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"openshift-service-ca.crt\"" Nov 03 18:55:17.203567 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.203540 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"kube-root-ca.crt\"" Nov 03 18:55:17.203728 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.203584 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9686c3\"/\"default-dockercfg-6l242\"" Nov 03 18:55:17.205489 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.205471 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl"] Nov 03 18:55:17.245925 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.245899 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-model-cache\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.246046 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.245952 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-home\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.246046 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.246032 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfjr\" (UniqueName: \"kubernetes.io/projected/b0f79162-e784-4c53-af8d-e7524a5cc763-kube-api-access-wpfjr\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.246129 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.246059 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-dshm\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.246129 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.246103 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f79162-e784-4c53-af8d-e7524a5cc763-tls-certs\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.246210 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.246197 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-tmp-dir\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.269143 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.269101 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:17.347109 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347075 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-tmp-dir\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347274 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347138 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-model-cache\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347274 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347193 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-home\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347274 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347218 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfjr\" (UniqueName: \"kubernetes.io/projected/b0f79162-e784-4c53-af8d-e7524a5cc763-kube-api-access-wpfjr\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347274 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347238 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-dshm\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347497 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347286 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f79162-e784-4c53-af8d-e7524a5cc763-tls-certs\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347574 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347543 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-tmp-dir\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347574 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347566 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-model-cache\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.347711 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.347674 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-home\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.349825 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.349793 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-dshm\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.349927 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.349811 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f79162-e784-4c53-af8d-e7524a5cc763-tls-certs\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.405285 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.405212 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfjr\" (UniqueName: \"kubernetes.io/projected/b0f79162-e784-4c53-af8d-e7524a5cc763-kube-api-access-wpfjr\") pod \"premium-w0-9686c3-kserve-6f89d849b7-h6bjl\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.419879 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.419855 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:17.802890 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.802862 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl"] Nov 03 18:55:17.805717 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:55:17.805670 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0f79162_e784_4c53_af8d_e7524a5cc763.slice/crio-0dbe644c4cb480e0d237fbf93818363af5c0041271dc569f34a8db336a2d0370 WatchSource:0}: Error finding container 0dbe644c4cb480e0d237fbf93818363af5c0041271dc569f34a8db336a2d0370: Status 404 returned error can't find the container with id 0dbe644c4cb480e0d237fbf93818363af5c0041271dc569f34a8db336a2d0370 Nov 03 18:55:17.916129 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.916095 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" event={"ID":"b0f79162-e784-4c53-af8d-e7524a5cc763","Type":"ContainerStarted","Data":"98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9"} Nov 03 18:55:17.916281 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:17.916138 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" event={"ID":"b0f79162-e784-4c53-af8d-e7524a5cc763","Type":"ContainerStarted","Data":"0dbe644c4cb480e0d237fbf93818363af5c0041271dc569f34a8db336a2d0370"} Nov 03 18:55:18.026438 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:18.026210 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podStartSLOduration=1.026192529 podStartE2EDuration="1.026192529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:18.026113937 +0000 UTC m=+992.907530040" watchObservedRunningTime="2026-11-03 18:55:18.026192529 +0000 UTC m=+992.907608632" Nov 03 18:55:18.280972 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:18.280938 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:55:18.420839 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:18.420811 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:19.712709 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:19.712665 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:24.405789 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:24.405703 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:25.964852 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:55:25.964811 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:55:27.420575 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:27.420545 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:27.433900 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:27.433871 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:27.994374 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:27.994344 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 18:55:29.061130 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:29.061092 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:30.810962 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.810930 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn"] Nov 03 18:55:30.816417 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.816394 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.865591 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.865567 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-fba266-openshift-default-dockercfg-dm45z\"" Nov 03 18:55:30.913695 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.913664 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn"] Nov 03 18:55:30.919732 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.919700 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.919858 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.919789 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.919858 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.919842 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.919977 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.919893 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.919977 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.919923 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.920087 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.919989 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw68d\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-kube-api-access-lw68d\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.920087 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.920032 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.920166 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.920148 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:30.920212 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:30.920190 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.021560 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021533 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.021560 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021564 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.021811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021585 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw68d\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-kube-api-access-lw68d\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.021811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021607 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.021811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021661 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.021811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021717 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.021811 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021786 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.022079 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021814 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.022079 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.021855 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.022079 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.022011 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.022227 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.022120 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.022227 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.022138 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.022227 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.022206 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.022506 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.022479 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.024056 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.024028 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.024280 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.024261 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.079337 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.079313 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.079475 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.079458 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw68d\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-kube-api-access-lw68d\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.129727 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.129706 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:31.557027 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.556951 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn"] Nov 03 18:55:31.607020 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:55:31.606990 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ec5ec5_d0de_4dd6_8f2d_164bdef7c420.slice/crio-e242203a2dfc0381c6bfb50e3abd81e18ff243432f1185d58210a322c8dcd5ec WatchSource:0}: Error finding container e242203a2dfc0381c6bfb50e3abd81e18ff243432f1185d58210a322c8dcd5ec: Status 404 returned error can't find the container with id e242203a2dfc0381c6bfb50e3abd81e18ff243432f1185d58210a322c8dcd5ec Nov 03 18:55:31.609370 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.609336 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:55:31.609466 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.609399 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:55:31.609466 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:31.609430 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:55:32.002891 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:32.002851 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" event={"ID":"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420","Type":"ContainerStarted","Data":"e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0"} Nov 03 18:55:32.003302 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:32.002946 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" event={"ID":"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420","Type":"ContainerStarted","Data":"e242203a2dfc0381c6bfb50e3abd81e18ff243432f1185d58210a322c8dcd5ec"} Nov 03 18:55:32.117806 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:32.117746 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podStartSLOduration=2.117731476 podStartE2EDuration="2.117731476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:55:32.115953982 +0000 UTC m=+1006.997370075" watchObservedRunningTime="2026-11-03 18:55:32.117731476 +0000 UTC m=+1006.999147578" Nov 03 18:55:32.130525 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:32.130487 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:32.131816 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:32.131788 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 18:55:32.131960 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:32.131852 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 18:55:33.130139 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:33.130108 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 18:55:33.130530 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:33.130169 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 18:55:33.625654 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:33.625619 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:34.130462 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:34.130430 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 18:55:34.130885 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:34.130483 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 18:55:35.130418 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:35.130385 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 18:55:35.130579 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:35.130433 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 18:55:36.130201 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:36.130167 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 18:55:36.130378 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:36.130235 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 18:55:37.130408 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:37.130369 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 18:55:37.130832 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:37.130443 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 18:55:38.299061 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:38.299022 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:38.299742 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:38.299718 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:38.300812 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:38.300792 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 18:55:38.337950 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:38.337918 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:42.844201 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:42.844165 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:46.141499 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:46.141463 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:47.547213 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:47.547173 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:50.546517 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:50.546487 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:52.135037 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:52.135002 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:55.936961 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:55:55.936918 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:55:56.725041 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:56.724998 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:55:59.989471 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:55:59.989437 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:56:01.345779 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:01.345743 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:06.333804 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:06.333770 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:11.231610 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:11.231573 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:15.827548 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:15.827505 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:20.411625 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:20.411582 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:25.203511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:25.203473 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:25.937556 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:56:25.937514 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:56:29.879277 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:29.879241 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:34.718551 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:34.718516 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:37.351110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:37.351068 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:37.558169 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:37.558133 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:39.316863 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:39.316802 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:43.944786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:43.944752 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:46.476959 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:46.476922 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:56:46.483781 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:46.483750 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:48.603824 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:48.603788 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:48.814814 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:48.814769 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:53.427453 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:53.427417 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:56:55.935760 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:56:55.935722 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:56:58.099554 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:56:58.099518 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:02.791399 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:02.791353 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:07.455246 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:07.455211 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:12.299620 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:12.299578 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:16.902128 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:16.902089 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:21.494389 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:21.494355 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:25.937714 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:57:25.937664 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:57:26.056396 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:26.056352 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:27.863226 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:27.863194 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:57:28.742086 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:28.742048 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd"] Nov 03 18:57:28.742286 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:28.742265 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" containerID="cri-o://94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd" gracePeriod=30 Nov 03 18:57:30.704797 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:30.704763 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:31.970998 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:31.970967 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt"] Nov 03 18:57:31.974639 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:31.974614 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.023826 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.023797 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2e052a-openshift-default-dockercfg-vjkcv\"" Nov 03 18:57:32.072748 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.072721 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt"] Nov 03 18:57:32.147950 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.147914 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148117 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148032 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148117 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148078 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148117 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148094 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148123 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148174 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148222 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148398 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148330 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.148398 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.148347 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gct4c\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-kube-api-access-gct4c\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.249818 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.249733 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.249818 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.249769 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.249818 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.249799 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.249917 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.249962 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250040 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250062 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gct4c\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-kube-api-access-gct4c\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250279 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250115 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250279 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250202 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250279 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250225 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250279 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250233 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250619 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250588 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250767 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250635 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.250819 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.250779 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.252393 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.252367 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.252393 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.252387 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.307615 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.307592 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.307756 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.307728 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gct4c\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-kube-api-access-gct4c\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:32.587864 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:32.587834 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:33.005970 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.005945 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt"] Nov 03 18:57:33.008549 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:57:33.008521 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fea62b_6fd4_40b2_8b7d_21b04f456fdd.slice/crio-f7557d852b052047faae293b91fd04233ea2f319600fa942e749552644418fc6 WatchSource:0}: Error finding container f7557d852b052047faae293b91fd04233ea2f319600fa942e749552644418fc6: Status 404 returned error can't find the container with id f7557d852b052047faae293b91fd04233ea2f319600fa942e749552644418fc6 Nov 03 18:57:33.011185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.011150 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:57:33.011318 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.011227 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:57:33.011318 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.011271 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:57:33.631137 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.631107 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p"] Nov 03 18:57:33.635276 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.635257 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.663452 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.663419 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" event={"ID":"96fea62b-6fd4-40b2-8b7d-21b04f456fdd","Type":"ContainerStarted","Data":"8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20"} Nov 03 18:57:33.663568 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.663472 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" event={"ID":"96fea62b-6fd4-40b2-8b7d-21b04f456fdd","Type":"ContainerStarted","Data":"f7557d852b052047faae293b91fd04233ea2f319600fa942e749552644418fc6"} Nov 03 18:57:33.732444 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.732401 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-3a7056-openshift-default-dockercfg-wjcwp\"" Nov 03 18:57:33.735380 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.735346 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p"] Nov 03 18:57:33.771384 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771300 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771590 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mttx\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-kube-api-access-2mttx\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771648 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb664c97-d248-4731-a27c-34b43cece904-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771805 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771875 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb664c97-d248-4731-a27c-34b43cece904-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771906 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771935 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.771972 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.780531 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.772027 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878362 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878295 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878538 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878411 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb664c97-d248-4731-a27c-34b43cece904-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878538 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878444 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878538 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878470 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878538 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878504 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878538 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878538 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878878 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878612 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878878 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878659 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mttx\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-kube-api-access-2mttx\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878878 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878712 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb664c97-d248-4731-a27c-34b43cece904-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.878878 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.878787 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.879098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.879067 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.879426 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.879400 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb664c97-d248-4731-a27c-34b43cece904-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.879598 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.879571 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.879910 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.879889 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.881241 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.881177 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb664c97-d248-4731-a27c-34b43cece904-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.881504 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.881482 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.936888 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.936855 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.937071 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.937020 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mttx\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-kube-api-access-2mttx\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:33.962177 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:33.949037 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:34.077245 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.076885 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:57:34.182286 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182190 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-credential-socket\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182286 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182254 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-podinfo\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182300 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-socket\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182351 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-data\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182400 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-certs\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182443 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-token\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-token\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182494 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b6678b-d97b-47f6-8698-8841a50d93a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b6678b-d97b-47f6-8698-8841a50d93a9-istiod-ca-cert\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182733 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182501 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-credential-socket" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:57:34.182733 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182521 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-kube-api-access-csq62\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-kube-api-access-csq62\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182733 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182555 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-socket" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:57:34.182733 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182615 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-envoy\") pod \"88b6678b-d97b-47f6-8698-8841a50d93a9\" (UID: \"88b6678b-d97b-47f6-8698-8841a50d93a9\") " Nov 03 18:57:34.182733 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182642 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-data" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:57:34.183015 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.182847 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-certs" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:57:34.183074 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.183026 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b6678b-d97b-47f6-8698-8841a50d93a9-istiod-ca-cert" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:57:34.183187 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.183071 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.183187 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.183086 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.183187 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.183096 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.183187 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.183105 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.184879 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.184850 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-token" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:34.185044 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.185022 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-envoy" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:57:34.185123 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.185022 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-kube-api-access-csq62" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "kube-api-access-csq62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:57:34.185123 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.185027 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-podinfo" pod "88b6678b-d97b-47f6-8698-8841a50d93a9" (UID: "88b6678b-d97b-47f6-8698-8841a50d93a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:57:34.192324 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.192277 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podStartSLOduration=3.19226625 podStartE2EDuration="3.19226625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:33.875108049 +0000 UTC m=+1128.756524151" watchObservedRunningTime="2026-11-03 18:57:34.19226625 +0000 UTC m=+1129.073682351" Nov 03 18:57:34.284463 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.284436 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.284463 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.284460 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.284611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.284470 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88b6678b-d97b-47f6-8698-8841a50d93a9-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.284611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.284479 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csq62\" (UniqueName: \"kubernetes.io/projected/88b6678b-d97b-47f6-8698-8841a50d93a9-kube-api-access-csq62\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.284611 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.284488 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88b6678b-d97b-47f6-8698-8841a50d93a9-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:57:34.396757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.396734 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p"] Nov 03 18:57:34.399051 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:57:34.399017 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb664c97_d248_4731_a27c_34b43cece904.slice/crio-c0c65b1e83d7ebb5e1a81727a05d1b0f530993ecbadc4d41252d9ef8e047bf78 WatchSource:0}: Error finding container c0c65b1e83d7ebb5e1a81727a05d1b0f530993ecbadc4d41252d9ef8e047bf78: Status 404 returned error can't find the container with id c0c65b1e83d7ebb5e1a81727a05d1b0f530993ecbadc4d41252d9ef8e047bf78 Nov 03 18:57:34.401653 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.401623 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:57:34.401778 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.401753 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:57:34.401845 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.401799 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:57:34.588333 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.588292 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:34.589607 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.589575 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 18:57:34.589761 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.589635 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 18:57:34.676162 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.676122 5988 generic.go:356] "Generic (PLEG): container finished" podID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerID="94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd" exitCode=0 Nov 03 18:57:34.676362 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.676180 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" Nov 03 18:57:34.676362 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.676242 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" event={"ID":"88b6678b-d97b-47f6-8698-8841a50d93a9","Type":"ContainerDied","Data":"94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd"} Nov 03 18:57:34.676362 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.676324 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd" event={"ID":"88b6678b-d97b-47f6-8698-8841a50d93a9","Type":"ContainerDied","Data":"fdb5db8fe00d5aa47a68ccaae22f5ffccc3522ab0fee99ae7eafc5f2b9817088"} Nov 03 18:57:34.676362 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.676360 5988 scope.go:122] "RemoveContainer" containerID="94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd" Nov 03 18:57:34.678506 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.678472 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" event={"ID":"bb664c97-d248-4731-a27c-34b43cece904","Type":"ContainerStarted","Data":"42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7"} Nov 03 18:57:34.678674 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.678533 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" event={"ID":"bb664c97-d248-4731-a27c-34b43cece904","Type":"ContainerStarted","Data":"c0c65b1e83d7ebb5e1a81727a05d1b0f530993ecbadc4d41252d9ef8e047bf78"} Nov 03 18:57:34.687433 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.687415 5988 scope.go:122] "RemoveContainer" containerID="94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd" Nov 03 18:57:34.687774 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:57:34.687752 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd\": container with ID starting with 94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd not found: ID does not exist" containerID="94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd" Nov 03 18:57:34.687845 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.687787 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd"} err="failed to get container status \"94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd\": rpc error: code = NotFound desc = could not find container \"94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd\": container with ID starting with 94777ec1586a63232fbab737e1609018556020a583619980402326d8889476dd not found: ID does not exist" Nov 03 18:57:34.842113 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.842050 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podStartSLOduration=1.8420313149999998 podStartE2EDuration="1.842031315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:34.841336762 +0000 UTC m=+1129.722752867" watchObservedRunningTime="2026-11-03 18:57:34.842031315 +0000 UTC m=+1129.723447417" Nov 03 18:57:34.964709 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.964557 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:34.979706 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.977868 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd"] Nov 03 18:57:34.979865 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:34.979732 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6t8sd"] Nov 03 18:57:35.306457 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:35.306362 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" path="/var/lib/kubelet/pods/88b6678b-d97b-47f6-8698-8841a50d93a9/volumes" Nov 03 18:57:35.339518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:35.339479 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:35.592736 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:35.591046 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 18:57:35.592736 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:35.591106 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 18:57:35.966763 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:35.965554 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 18:57:35.966763 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:35.965616 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 18:57:36.593707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:36.593029 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 18:57:36.593707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:36.593092 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 18:57:36.796645 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:36.796281 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:36.964228 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:36.964128 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 18:57:36.964507 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:36.964464 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 18:57:37.591035 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:37.590998 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 18:57:37.591344 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:37.591315 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 18:57:37.953762 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:37.949987 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 18:57:37.953762 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:37.950049 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 18:57:38.589708 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:38.589191 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 18:57:38.589708 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:38.589269 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 18:57:38.950106 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:38.950011 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 18:57:38.950366 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:38.950333 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 18:57:39.588735 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:39.588679 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 18:57:39.589120 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:39.588772 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 18:57:39.951566 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:39.951049 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 18:57:39.951566 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:39.951115 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 18:57:40.204018 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:40.203923 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:40.415365 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:40.415334 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:40.594057 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:40.594020 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:40.594544 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:40.594428 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:40.595281 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:40.595259 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 18:57:40.950296 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:40.950205 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 18:57:40.950296 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:40.950255 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 18:57:41.954366 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:41.954335 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:41.954809 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:41.954623 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:41.955220 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:41.955204 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 18:57:44.813901 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:44.813862 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:46.042157 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:46.042121 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:46.838217 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:46.838184 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:49.676540 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:49.676499 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:54.322454 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:54.322419 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:55.936192 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:57:55.936153 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:57:58.918239 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:58.918197 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:59.473081 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.473046 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646"] Nov 03 18:57:59.473564 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.473531 5988 cpu_manager.go:414] "containerMap: removing container" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" Nov 03 18:57:59.473564 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.473561 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" Nov 03 18:57:59.473761 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.473660 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b6678b-d97b-47f6-8698-8841a50d93a9" containerName="istio-proxy" Nov 03 18:57:59.478489 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.478467 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.574258 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.574228 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-355e1a\"/\"distinct-w1-355e1a-kserve-self-signed-certs\"" Nov 03 18:57:59.576304 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.576276 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646"] Nov 03 18:57:59.632338 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.632306 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:57:59.671998 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.671970 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-model-cache\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.672144 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.672019 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tls-certs\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.672226 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.672201 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbc5k\" (UniqueName: \"kubernetes.io/projected/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-kube-api-access-kbc5k\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.672266 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.672255 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-dshm\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.672332 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.672318 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-home\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.672413 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.672397 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tmp-dir\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773139 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773045 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tls-certs\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773168 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbc5k\" (UniqueName: \"kubernetes.io/projected/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-kube-api-access-kbc5k\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773207 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-dshm\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773303 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773267 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-home\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773478 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773436 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tmp-dir\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773734 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773710 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-model-cache\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773890 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773744 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-home\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.773890 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.773828 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tmp-dir\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.774064 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.774041 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-model-cache\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.775857 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.775836 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-dshm\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.776224 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.776206 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tls-certs\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:57:59.833136 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:57:59.833102 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbc5k\" (UniqueName: \"kubernetes.io/projected/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-kube-api-access-kbc5k\") pod \"distinct-w1-355e1a-kserve-6699c7796-pr646\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:58:00.089937 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:00.089908 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:58:00.457707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:00.457611 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646"] Nov 03 18:58:00.461179 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:58:00.461146 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae6b35a7_0100_41e8_ad9c_d4e5efd0120b.slice/crio-2575988b05cde4ccb74b7e0865594be489b00a2ecaf2e503d14841f7d7187f0e WatchSource:0}: Error finding container 2575988b05cde4ccb74b7e0865594be489b00a2ecaf2e503d14841f7d7187f0e: Status 404 returned error can't find the container with id 2575988b05cde4ccb74b7e0865594be489b00a2ecaf2e503d14841f7d7187f0e Nov 03 18:58:00.833349 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:00.833312 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" event={"ID":"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b","Type":"ContainerStarted","Data":"af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3"} Nov 03 18:58:00.833504 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:00.833360 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" event={"ID":"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b","Type":"ContainerStarted","Data":"2575988b05cde4ccb74b7e0865594be489b00a2ecaf2e503d14841f7d7187f0e"} Nov 03 18:58:00.945094 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:00.945050 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podStartSLOduration=1.945036015 podStartE2EDuration="1.945036015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:00.943909471 +0000 UTC m=+1155.825325573" watchObservedRunningTime="2026-11-03 18:58:00.945036015 +0000 UTC m=+1155.826452169" Nov 03 18:58:01.090449 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:01.090370 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:58:01.339511 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:01.339472 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:58:03.508734 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:03.508703 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:08.130538 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:08.130498 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:08.977284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:08.977246 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp"] Nov 03 18:58:08.985279 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:08.985239 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.071834 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.071790 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10baada4-97eb-4266-b663-ebd01cd83a23-tls-certs\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.072019 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.071848 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-model-cache\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.072019 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.071948 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-dshm\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.072110 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.072055 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p299p\" (UniqueName: \"kubernetes.io/projected/10baada4-97eb-4266-b663-ebd01cd83a23-kube-api-access-p299p\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.072171 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.072123 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-tmp-dir\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.072252 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.072234 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-home\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.081369 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.081341 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-fba266\"/\"distinct-w6-fba266-kserve-self-signed-certs\"" Nov 03 18:58:09.083348 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.083317 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp"] Nov 03 18:58:09.143029 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.142993 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:09.173556 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.173507 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-tmp-dir\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.173762 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.173604 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-home\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.173762 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.173655 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10baada4-97eb-4266-b663-ebd01cd83a23-tls-certs\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.173762 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.173739 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-model-cache\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.173940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.173771 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-dshm\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.173940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.173881 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p299p\" (UniqueName: \"kubernetes.io/projected/10baada4-97eb-4266-b663-ebd01cd83a23-kube-api-access-p299p\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.174053 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.174020 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-tmp-dir\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.174130 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.174105 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-home\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.174185 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.174163 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-model-cache\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.176339 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.176298 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-dshm\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.176454 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.176404 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10baada4-97eb-4266-b663-ebd01cd83a23-tls-certs\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.232081 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.232002 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p299p\" (UniqueName: \"kubernetes.io/projected/10baada4-97eb-4266-b663-ebd01cd83a23-kube-api-access-p299p\") pod \"distinct-w6-fba266-kserve-7bbbc6448-l95sp\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.298518 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.298474 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:09.684974 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.684949 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp"] Nov 03 18:58:09.687862 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:58:09.687829 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10baada4_97eb_4266_b663_ebd01cd83a23.slice/crio-05d25f01f5ab2309c69d2289c94e98af37186a712b698158a36b269daf1e253d WatchSource:0}: Error finding container 05d25f01f5ab2309c69d2289c94e98af37186a712b698158a36b269daf1e253d: Status 404 returned error can't find the container with id 05d25f01f5ab2309c69d2289c94e98af37186a712b698158a36b269daf1e253d Nov 03 18:58:09.886816 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.886780 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" event={"ID":"10baada4-97eb-4266-b663-ebd01cd83a23","Type":"ContainerStarted","Data":"074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251"} Nov 03 18:58:09.886997 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.886832 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" event={"ID":"10baada4-97eb-4266-b663-ebd01cd83a23","Type":"ContainerStarted","Data":"05d25f01f5ab2309c69d2289c94e98af37186a712b698158a36b269daf1e253d"} Nov 03 18:58:09.998325 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:09.998218 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podStartSLOduration=1.998202471 podStartE2EDuration="1.998202471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:09.996154178 +0000 UTC m=+1164.877570283" watchObservedRunningTime="2026-11-03 18:58:09.998202471 +0000 UTC m=+1164.879618572" Nov 03 18:58:10.090751 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:10.090718 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:58:10.106011 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:10.105981 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:58:10.299437 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:10.299346 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:10.903594 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:10.903564 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 18:58:12.698237 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:12.698199 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:17.306514 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:17.306477 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:19.299436 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:19.299404 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:19.311626 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:19.311595 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:19.953889 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:19.953859 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 18:58:21.904823 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:21.904775 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:22.342696 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:22.342655 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:25.938831 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:58:25.938736 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:58:26.607749 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:26.607718 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:31.200597 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:31.200566 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:33.818036 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.818004 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx"] Nov 03 18:58:33.821921 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.821900 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.847490 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.847240 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-tmp-dir\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.847490 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.847284 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-model-cache\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.847490 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.847336 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-home\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.847768 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.847558 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2667b76-561f-4f80-8338-04fb19a5a99a-tls-certs\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.847768 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.847601 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-dshm\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.847768 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.847631 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84cq\" (UniqueName: \"kubernetes.io/projected/c2667b76-561f-4f80-8338-04fb19a5a99a-kube-api-access-d84cq\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.917258 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.917226 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"openshift-service-ca.crt\"" Nov 03 18:58:33.917401 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.917308 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"kube-root-ca.crt\"" Nov 03 18:58:33.917401 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.917341 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"distinct-w2-32a59b-kserve-self-signed-certs\"" Nov 03 18:58:33.917401 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.917374 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-32a59b\"/\"default-dockercfg-zbhk5\"" Nov 03 18:58:33.919217 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.919196 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx"] Nov 03 18:58:33.948780 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.948751 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2667b76-561f-4f80-8338-04fb19a5a99a-tls-certs\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.948916 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.948798 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-dshm\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.948916 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.948825 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84cq\" (UniqueName: \"kubernetes.io/projected/c2667b76-561f-4f80-8338-04fb19a5a99a-kube-api-access-d84cq\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.948916 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.948907 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-tmp-dir\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.949062 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.948936 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-model-cache\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.949062 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.948986 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-home\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.949301 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.949272 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-tmp-dir\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.949449 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.949429 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-model-cache\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.949548 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.949515 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-home\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.951099 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.951079 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-dshm\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.951295 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.951277 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2667b76-561f-4f80-8338-04fb19a5a99a-tls-certs\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:33.965247 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:33.965208 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:34.008390 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:34.008363 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84cq\" (UniqueName: \"kubernetes.io/projected/c2667b76-561f-4f80-8338-04fb19a5a99a-kube-api-access-d84cq\") pod \"distinct-w2-32a59b-kserve-77569f9cf9-t48sx\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:34.134696 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:34.134603 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:34.522029 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:34.522006 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx"] Nov 03 18:58:34.524590 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:58:34.524561 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2667b76_561f_4f80_8338_04fb19a5a99a.slice/crio-e755c1609fb58cafa00c7b2f606b4ecdf48a69e7b7deeb1aa0fee652abf04ea8 WatchSource:0}: Error finding container e755c1609fb58cafa00c7b2f606b4ecdf48a69e7b7deeb1aa0fee652abf04ea8: Status 404 returned error can't find the container with id e755c1609fb58cafa00c7b2f606b4ecdf48a69e7b7deeb1aa0fee652abf04ea8 Nov 03 18:58:35.025759 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:35.025721 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" event={"ID":"c2667b76-561f-4f80-8338-04fb19a5a99a","Type":"ContainerStarted","Data":"5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0"} Nov 03 18:58:35.025759 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:35.025760 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" event={"ID":"c2667b76-561f-4f80-8338-04fb19a5a99a","Type":"ContainerStarted","Data":"e755c1609fb58cafa00c7b2f606b4ecdf48a69e7b7deeb1aa0fee652abf04ea8"} Nov 03 18:58:35.134696 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:35.134658 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:35.137585 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:35.137538 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podStartSLOduration=2.137525901 podStartE2EDuration="2.137525901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:35.13575255 +0000 UTC m=+1190.017168655" watchObservedRunningTime="2026-11-03 18:58:35.137525901 +0000 UTC m=+1190.018942002" Nov 03 18:58:35.823835 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:35.823795 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:37.842601 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:37.842564 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:58:40.413208 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:40.413164 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:44.135256 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:44.135223 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:44.147598 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:44.147573 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:45.021658 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:45.021615 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:45.089875 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:45.089845 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 18:58:47.191727 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:47.191693 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:48.271167 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.271134 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57"] Nov 03 18:58:48.276378 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.276356 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.334755 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.334727 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-355e1a-openshift-default-dockercfg-7nkhf\"" Nov 03 18:58:48.373606 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.373579 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57"] Nov 03 18:58:48.381826 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.381799 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpdm\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-kube-api-access-dgpdm\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.381946 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.381861 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.382009 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.381985 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.382066 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.382038 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.382161 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.382146 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.382277 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.382197 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.382328 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.382280 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.382328 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.382317 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.382401 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.382351 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483605 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483578 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483613 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483635 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483673 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483786 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483727 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpdm\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-kube-api-access-dgpdm\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483800 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483862 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.483943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483902 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.484048 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.483955 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.484284 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.484258 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.484414 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.484321 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.484482 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.484409 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.484482 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.484429 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.484482 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.484451 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.486016 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.485986 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.486276 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.486258 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.541440 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.541379 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.541541 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.541434 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpdm\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-kube-api-access-dgpdm\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:48.590086 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:48.590062 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:49.008946 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.008918 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57"] Nov 03 18:58:49.010940 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:58:49.010911 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afd215e_7b95_4ef7_b1d5_1ab1c58764e9.slice/crio-0c9669e6cffd6d2eb07f344319464d2f11a5bb87e16dd66d985246de30413093 WatchSource:0}: Error finding container 0c9669e6cffd6d2eb07f344319464d2f11a5bb87e16dd66d985246de30413093: Status 404 returned error can't find the container with id 0c9669e6cffd6d2eb07f344319464d2f11a5bb87e16dd66d985246de30413093 Nov 03 18:58:49.013491 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.013459 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:58:49.013588 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.013563 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:58:49.013655 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.013611 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:58:49.105356 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.105318 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" event={"ID":"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9","Type":"ContainerStarted","Data":"785f789f233fdba23bfec20b0c37a2ebac322b8d6a623b5ef6dc7bb20c8a45a1"} Nov 03 18:58:49.105609 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.105578 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" event={"ID":"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9","Type":"ContainerStarted","Data":"0c9669e6cffd6d2eb07f344319464d2f11a5bb87e16dd66d985246de30413093"} Nov 03 18:58:49.219952 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.219904 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podStartSLOduration=1.219890085 podStartE2EDuration="1.219890085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:49.218511908 +0000 UTC m=+1204.099928022" watchObservedRunningTime="2026-11-03 18:58:49.219890085 +0000 UTC m=+1204.101306186" Nov 03 18:58:49.591334 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.591302 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:49.606983 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.606952 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:49.872587 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:49.872491 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:50.592456 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:50.592417 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 18:58:50.592958 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:50.592490 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 18:58:51.590589 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:51.590550 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 18:58:51.590788 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:51.590606 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 18:58:52.591105 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:52.591065 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 18:58:52.591530 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:52.591131 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 18:58:53.590491 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:53.590451 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 18:58:53.590694 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:53.590516 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 18:58:54.308628 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:54.308591 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:58:54.590805 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:54.590774 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 18:58:54.591002 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:54.590829 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 18:58:55.591704 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:55.591645 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 18:58:55.592106 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:55.591750 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 18:58:55.943430 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:58:55.943331 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:58:56.595363 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:56.595332 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:56.595760 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:56.595659 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:56.596348 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:56.596329 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 18:58:58.894772 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:58:58.894736 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:03.503599 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:03.503566 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:07.493215 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:07.493179 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:59:08.105352 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:08.105317 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:12.710162 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:12.710123 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:14.151248 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:14.151216 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:17.320923 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:17.320894 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:19.118366 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.118333 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd"] Nov 03 18:59:19.122427 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.122405 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.171477 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.171449 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-32a59b-openshift-default-dockercfg-rm7v5\"" Nov 03 18:59:19.219823 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.219794 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd"] Nov 03 18:59:19.238562 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.238541 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.238701 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.238600 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.238747 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.238713 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.238783 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.238745 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.238783 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.238762 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.238940 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.238918 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8pw\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-kube-api-access-cc8pw\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.238999 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.238971 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.239043 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.239007 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.239096 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.239064 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.339981 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.339956 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8pw\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-kube-api-access-cc8pw\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340112 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.339991 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340112 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340048 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340112 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340083 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340262 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340120 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340262 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340192 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340262 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340240 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340415 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340262 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340415 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340282 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340415 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340369 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340581 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340450 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340581 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340557 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.340705 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.340655 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.341082 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.341061 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.342314 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.342294 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.342453 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.342435 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.411911 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.411852 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8pw\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-kube-api-access-cc8pw\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.411911 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.411875 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.437791 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.437767 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:19.861153 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.861113 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd"] Nov 03 18:59:19.862537 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:59:19.862503 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4adea64_c229_4ae9_9ee1_bb740a9cad41.slice/crio-2d8b8480a7af13235f8e1d4ffe3609802905eb3150e05c18529b80ce61fcace5 WatchSource:0}: Error finding container 2d8b8480a7af13235f8e1d4ffe3609802905eb3150e05c18529b80ce61fcace5: Status 404 returned error can't find the container with id 2d8b8480a7af13235f8e1d4ffe3609802905eb3150e05c18529b80ce61fcace5 Nov 03 18:59:19.864943 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.864915 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:59:19.865007 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.864976 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:59:19.865040 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:19.865004 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:59:20.276591 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:20.276514 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" event={"ID":"f4adea64-c229-4ae9-9ee1-bb740a9cad41","Type":"ContainerStarted","Data":"3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441"} Nov 03 18:59:20.276591 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:20.276563 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" event={"ID":"f4adea64-c229-4ae9-9ee1-bb740a9cad41","Type":"ContainerStarted","Data":"2d8b8480a7af13235f8e1d4ffe3609802905eb3150e05c18529b80ce61fcace5"} Nov 03 18:59:20.392461 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:20.392412 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podStartSLOduration=1.392396346 podStartE2EDuration="1.392396346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:20.391137387 +0000 UTC m=+1235.272553524" watchObservedRunningTime="2026-11-03 18:59:20.392396346 +0000 UTC m=+1235.273812452" Nov 03 18:59:20.437937 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:20.437908 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:20.439830 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:20.439803 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:59:20.439990 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:20.439855 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 18:59:21.438239 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:21.438205 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:59:21.438707 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:21.438256 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 18:59:21.949710 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:21.945970 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:22.438950 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:22.438915 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:59:22.439401 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:22.438981 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 18:59:23.438658 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:23.438622 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:59:23.438855 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:23.438670 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 18:59:24.438485 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:24.438450 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:59:24.438915 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:24.438511 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 18:59:25.438184 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:25.438149 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:59:25.438368 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:25.438196 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 18:59:25.936404 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:59:25.936370 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:59:26.438442 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:26.438396 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 18:59:26.438621 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:26.438464 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 18:59:26.543757 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:26.543718 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:27.442963 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:27.442932 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:27.443438 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:27.443337 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:27.444099 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:27.444084 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 18:59:31.058623 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:31.058586 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:33.289712 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.289648 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh"] Nov 03 18:59:33.293869 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.293846 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.342924 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.342884 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a4cacd-openshift-default-dockercfg-hkxvc\"" Nov 03 18:59:33.392367 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.392326 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh"] Nov 03 18:59:33.421141 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421100 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.421341 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421176 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.421426 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421401 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.421478 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421454 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.421555 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421542 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.421603 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421568 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.421659 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421610 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.421805 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.421788 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqz7k\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-kube-api-access-gqz7k\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.422152 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.422127 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.522853 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.522818 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523061 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.522905 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqz7k\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-kube-api-access-gqz7k\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523061 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.522980 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523061 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523014 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523061 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523046 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523283 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523080 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523283 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523123 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523283 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523192 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523283 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523215 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523567 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523537 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523647 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523566 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523763 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523663 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.523763 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.523725 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.524134 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.524108 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.525438 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.525414 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.525895 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.525879 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.581187 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.581153 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.581354 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.581192 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqz7k\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-kube-api-access-gqz7k\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:33.607975 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:33.607942 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:34.032134 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.032109 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh"] Nov 03 18:59:34.034197 ip-10-0-139-161 kubenswrapper[5988]: W1103 18:59:34.034161 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2377db91_edd4_4b77_b952_6c2eb8ac1f26.slice/crio-441d3460533d51f939eeb21a7b940fb697d04190b5aa3cba0a285220ca0cb7b3 WatchSource:0}: Error finding container 441d3460533d51f939eeb21a7b940fb697d04190b5aa3cba0a285220ca0cb7b3: Status 404 returned error can't find the container with id 441d3460533d51f939eeb21a7b940fb697d04190b5aa3cba0a285220ca0cb7b3 Nov 03 18:59:34.036442 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.036411 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:59:34.036504 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.036489 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:59:34.036549 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.036523 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 18:59:34.365712 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.365652 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" event={"ID":"2377db91-edd4-4b77-b952-6c2eb8ac1f26","Type":"ContainerStarted","Data":"2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e"} Nov 03 18:59:34.366122 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.365738 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" event={"ID":"2377db91-edd4-4b77-b952-6c2eb8ac1f26","Type":"ContainerStarted","Data":"441d3460533d51f939eeb21a7b940fb697d04190b5aa3cba0a285220ca0cb7b3"} Nov 03 18:59:34.484098 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.484039 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podStartSLOduration=1.484023477 podStartE2EDuration="1.484023477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:34.481882339 +0000 UTC m=+1249.363298442" watchObservedRunningTime="2026-11-03 18:59:34.484023477 +0000 UTC m=+1249.365439637" Nov 03 18:59:34.610470 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:34.610044 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:35.610305 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:35.610259 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:59:35.610982 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:35.610325 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:59:35.628819 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:35.628787 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:36.608513 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:36.608473 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:59:36.608721 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:36.608529 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:59:37.608417 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:37.608376 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:59:37.608928 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:37.608439 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:59:38.608755 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:38.608721 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:59:38.609135 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:38.608779 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:59:39.609252 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:39.609216 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:59:39.609669 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:39.609268 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:59:40.420197 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:40.420143 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:40.608244 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:40.608200 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:59:40.608439 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:40.608266 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:59:41.613154 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:41.613125 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:41.613553 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:41.613394 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:41.614223 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:41.614206 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 18:59:45.001317 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:45.001276 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:47.585931 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:47.585899 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:49.634650 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:49.634618 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:49.966878 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:49.966799 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 18:59:54.216929 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:54.216845 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 18:59:55.937604 ip-10-0-139-161 kubenswrapper[5988]: E1103 18:59:55.937557 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 18:59:58.818871 ip-10-0-139-161 kubenswrapper[5988]: I1103 18:59:58.818836 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:00.179233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.179199 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk"] Nov 03 19:00:00.184596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.184575 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.198257 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.198228 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz"] Nov 03 19:00:00.202131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.202110 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.233409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.233375 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:00:00.277808 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.277771 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk"] Nov 03 19:00:00.311272 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.311234 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz"] Nov 03 19:00:00.350131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.350096 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3abfb10-50a4-49d4-b6d9-8956892d3615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.350277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.350211 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e019be6-cad6-4c8e-ade1-5252d051ee2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.350277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.350264 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8pq\" (UniqueName: \"kubernetes.io/projected/b3abfb10-50a4-49d4-b6d9-8956892d3615-kube-api-access-fq8pq\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.350411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.350391 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxclv\" (UniqueName: \"kubernetes.io/projected/6e019be6-cad6-4c8e-ade1-5252d051ee2a-kube-api-access-mxclv\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.451158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.451065 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxclv\" (UniqueName: \"kubernetes.io/projected/6e019be6-cad6-4c8e-ade1-5252d051ee2a-kube-api-access-mxclv\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.451328 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.451196 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3abfb10-50a4-49d4-b6d9-8956892d3615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.451328 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.451280 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e019be6-cad6-4c8e-ade1-5252d051ee2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.451328 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.451322 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8pq\" (UniqueName: \"kubernetes.io/projected/b3abfb10-50a4-49d4-b6d9-8956892d3615-kube-api-access-fq8pq\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.452052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.452030 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e019be6-cad6-4c8e-ade1-5252d051ee2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.452115 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.452081 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3abfb10-50a4-49d4-b6d9-8956892d3615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.510254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.510213 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxclv\" (UniqueName: \"kubernetes.io/projected/6e019be6-cad6-4c8e-ade1-5252d051ee2a-kube-api-access-mxclv\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.510419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.510287 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8pq\" (UniqueName: \"kubernetes.io/projected/b3abfb10-50a4-49d4-b6d9-8956892d3615-kube-api-access-fq8pq\") pod \"maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.515071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.515053 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:00.797636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.797560 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:00.884370 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.884331 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz"] Nov 03 19:00:00.890008 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:00:00.889976 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3abfb10_50a4_49d4_b6d9_8956892d3615.slice/crio-8cdc90c725850a087e0f42a4c85abc58e615212053a449960769e3924816e7d7 WatchSource:0}: Error finding container 8cdc90c725850a087e0f42a4c85abc58e615212053a449960769e3924816e7d7: Status 404 returned error can't find the container with id 8cdc90c725850a087e0f42a4c85abc58e615212053a449960769e3924816e7d7 Nov 03 19:00:00.892334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:00.892311 5988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:00:01.163844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:01.163813 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk"] Nov 03 19:00:01.166568 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:00:01.166537 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e019be6_cad6_4c8e_ade1_5252d051ee2a.slice/crio-6d5de2dfa2758122740e16fafdb6a58b3715195395008e5425200b0dc3e478af WatchSource:0}: Error finding container 6d5de2dfa2758122740e16fafdb6a58b3715195395008e5425200b0dc3e478af: Status 404 returned error can't find the container with id 6d5de2dfa2758122740e16fafdb6a58b3715195395008e5425200b0dc3e478af Nov 03 19:00:01.528406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:01.528316 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" event={"ID":"6e019be6-cad6-4c8e-ade1-5252d051ee2a","Type":"ContainerStarted","Data":"6d5de2dfa2758122740e16fafdb6a58b3715195395008e5425200b0dc3e478af"} Nov 03 19:00:01.529565 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:01.529542 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" event={"ID":"b3abfb10-50a4-49d4-b6d9-8956892d3615","Type":"ContainerStarted","Data":"8cdc90c725850a087e0f42a4c85abc58e615212053a449960769e3924816e7d7"} Nov 03 19:00:03.597502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:03.597470 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:05.656047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:05.656013 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:08.288430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:08.288398 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:09.578796 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:09.578751 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" event={"ID":"6e019be6-cad6-4c8e-ade1-5252d051ee2a","Type":"ContainerStarted","Data":"18966fa4c51bfa59fb00ab55f7c6c443d4bb6e4f4c84a19658b5024f13914330"} Nov 03 19:00:09.580463 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:09.580434 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" event={"ID":"b3abfb10-50a4-49d4-b6d9-8956892d3615","Type":"ContainerStarted","Data":"2c3d56a7b4ec1ffbfc5470c09d9ff3d551ee60553802bba83d2c4c5dea901293"} Nov 03 19:00:09.687918 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:09.687864 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" podStartSLOduration=1.988829636 podStartE2EDuration="9.687847389s" totalImagesPullingTime="7.699017753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:09.687281867 +0000 UTC m=+1284.568697972" watchObservedRunningTime="2026-11-03 19:00:09.687847389 +0000 UTC m=+1284.569263481" Nov 03 19:00:09.800188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:09.800125 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" podStartSLOduration=1.5135676679999999 podStartE2EDuration="9.800105331s" totalImagesPullingTime="8.286537663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:09.797326341 +0000 UTC m=+1284.678742443" watchObservedRunningTime="2026-11-03 19:00:09.800105331 +0000 UTC m=+1284.681521434" Nov 03 19:00:10.587188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:10.587157 5988 generic.go:356] "Generic (PLEG): container finished" podID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerID="2c3d56a7b4ec1ffbfc5470c09d9ff3d551ee60553802bba83d2c4c5dea901293" exitCode=60 Nov 03 19:00:10.587634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:10.587254 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" event={"ID":"b3abfb10-50a4-49d4-b6d9-8956892d3615","Type":"ContainerDied","Data":"2c3d56a7b4ec1ffbfc5470c09d9ff3d551ee60553802bba83d2c4c5dea901293"} Nov 03 19:00:10.587634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:10.587540 5988 scope.go:122] "RemoveContainer" containerID="2c3d56a7b4ec1ffbfc5470c09d9ff3d551ee60553802bba83d2c4c5dea901293" Nov 03 19:00:10.589020 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:10.588995 5988 generic.go:356] "Generic (PLEG): container finished" podID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerID="18966fa4c51bfa59fb00ab55f7c6c443d4bb6e4f4c84a19658b5024f13914330" exitCode=60 Nov 03 19:00:10.589131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:10.589022 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" event={"ID":"6e019be6-cad6-4c8e-ade1-5252d051ee2a","Type":"ContainerDied","Data":"18966fa4c51bfa59fb00ab55f7c6c443d4bb6e4f4c84a19658b5024f13914330"} Nov 03 19:00:10.589348 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:10.589330 5988 scope.go:122] "RemoveContainer" containerID="18966fa4c51bfa59fb00ab55f7c6c443d4bb6e4f4c84a19658b5024f13914330" Nov 03 19:00:10.836103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:10.836068 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:11.596601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.596572 5988 generic.go:356] "Generic (PLEG): container finished" podID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerID="78435bce3f36e2b063a50269ba5d444de13492c64d4c0891b420dada3e11cdb9" exitCode=60 Nov 03 19:00:11.597090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.596654 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" event={"ID":"6e019be6-cad6-4c8e-ade1-5252d051ee2a","Type":"ContainerDied","Data":"78435bce3f36e2b063a50269ba5d444de13492c64d4c0891b420dada3e11cdb9"} Nov 03 19:00:11.597090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.596741 5988 scope.go:122] "RemoveContainer" containerID="18966fa4c51bfa59fb00ab55f7c6c443d4bb6e4f4c84a19658b5024f13914330" Nov 03 19:00:11.597090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.597051 5988 scope.go:122] "RemoveContainer" containerID="78435bce3f36e2b063a50269ba5d444de13492c64d4c0891b420dada3e11cdb9" Nov 03 19:00:11.597364 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:11.597344 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk_odh-ai-gateway-infra(6e019be6-cad6-4c8e-ade1-5252d051ee2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" Nov 03 19:00:11.598967 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.598943 5988 generic.go:356] "Generic (PLEG): container finished" podID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerID="2981dfe78bfd8f1b2dcfc56aeb5710ec989f86d4916c9a486d9b19ff3ba3baa9" exitCode=60 Nov 03 19:00:11.599067 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.599020 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" event={"ID":"b3abfb10-50a4-49d4-b6d9-8956892d3615","Type":"ContainerDied","Data":"2981dfe78bfd8f1b2dcfc56aeb5710ec989f86d4916c9a486d9b19ff3ba3baa9"} Nov 03 19:00:11.599227 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.599215 5988 scope.go:122] "RemoveContainer" containerID="2981dfe78bfd8f1b2dcfc56aeb5710ec989f86d4916c9a486d9b19ff3ba3baa9" Nov 03 19:00:11.599435 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:11.599419 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz_odh-ai-gateway-infra(b3abfb10-50a4-49d4-b6d9-8956892d3615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" Nov 03 19:00:11.617438 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:11.617418 5988 scope.go:122] "RemoveContainer" containerID="2c3d56a7b4ec1ffbfc5470c09d9ff3d551ee60553802bba83d2c4c5dea901293" Nov 03 19:00:12.608310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:12.608284 5988 scope.go:122] "RemoveContainer" containerID="2981dfe78bfd8f1b2dcfc56aeb5710ec989f86d4916c9a486d9b19ff3ba3baa9" Nov 03 19:00:12.608811 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:12.608644 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz_odh-ai-gateway-infra(b3abfb10-50a4-49d4-b6d9-8956892d3615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" Nov 03 19:00:12.610136 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:12.610113 5988 scope.go:122] "RemoveContainer" containerID="78435bce3f36e2b063a50269ba5d444de13492c64d4c0891b420dada3e11cdb9" Nov 03 19:00:12.610395 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:12.610374 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk_odh-ai-gateway-infra(6e019be6-cad6-4c8e-ade1-5252d051ee2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" Nov 03 19:00:12.919217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:12.919130 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:16.073794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:16.073756 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:17.621952 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:17.621907 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:21.927639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:21.927582 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:22.250001 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:22.249861 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:22.582757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:22.582720 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:00:23.268087 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:23.268054 5988 scope.go:122] "RemoveContainer" containerID="78435bce3f36e2b063a50269ba5d444de13492c64d4c0891b420dada3e11cdb9" Nov 03 19:00:23.673931 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:23.673889 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" event={"ID":"6e019be6-cad6-4c8e-ade1-5252d051ee2a","Type":"ContainerStarted","Data":"f1ce151f57f6e4900bde4a57041536c5364d5443fd5903c249c44e68fd2685ee"} Nov 03 19:00:24.684621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:24.684522 5988 generic.go:356] "Generic (PLEG): container finished" podID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerID="f1ce151f57f6e4900bde4a57041536c5364d5443fd5903c249c44e68fd2685ee" exitCode=60 Nov 03 19:00:24.684621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:24.684562 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" event={"ID":"6e019be6-cad6-4c8e-ade1-5252d051ee2a","Type":"ContainerDied","Data":"f1ce151f57f6e4900bde4a57041536c5364d5443fd5903c249c44e68fd2685ee"} Nov 03 19:00:24.684621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:24.684618 5988 scope.go:122] "RemoveContainer" containerID="78435bce3f36e2b063a50269ba5d444de13492c64d4c0891b420dada3e11cdb9" Nov 03 19:00:24.685290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:24.685127 5988 scope.go:122] "RemoveContainer" containerID="f1ce151f57f6e4900bde4a57041536c5364d5443fd5903c249c44e68fd2685ee" Nov 03 19:00:24.685499 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:24.685469 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk_odh-ai-gateway-infra(6e019be6-cad6-4c8e-ade1-5252d051ee2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" Nov 03 19:00:24.792308 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:24.792269 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk"] Nov 03 19:00:25.269640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.269610 5988 scope.go:122] "RemoveContainer" containerID="2981dfe78bfd8f1b2dcfc56aeb5710ec989f86d4916c9a486d9b19ff3ba3baa9" Nov 03 19:00:25.692776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.692739 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" event={"ID":"b3abfb10-50a4-49d4-b6d9-8956892d3615","Type":"ContainerStarted","Data":"e15e3fac6a561dd3bede4f1ff980f30c720580630828b72d795aac6133ec6b58"} Nov 03 19:00:25.854894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.854866 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:25.943405 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:25.943366 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:00:25.962013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.961983 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e019be6-cad6-4c8e-ade1-5252d051ee2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e019be6-cad6-4c8e-ade1-5252d051ee2a-maas-api-service-ca\") pod \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " Nov 03 19:00:25.962204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.962058 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e019be6-cad6-4c8e-ade1-5252d051ee2a-kube-api-access-mxclv\" (UniqueName: \"kubernetes.io/projected/6e019be6-cad6-4c8e-ade1-5252d051ee2a-kube-api-access-mxclv\") pod \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\" (UID: \"6e019be6-cad6-4c8e-ade1-5252d051ee2a\") " Nov 03 19:00:25.962437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.962381 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e019be6-cad6-4c8e-ade1-5252d051ee2a-maas-api-service-ca" pod "6e019be6-cad6-4c8e-ade1-5252d051ee2a" (UID: "6e019be6-cad6-4c8e-ade1-5252d051ee2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:25.962621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.962601 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e019be6-cad6-4c8e-ade1-5252d051ee2a-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:25.964217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:25.964189 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e019be6-cad6-4c8e-ade1-5252d051ee2a-kube-api-access-mxclv" pod "6e019be6-cad6-4c8e-ade1-5252d051ee2a" (UID: "6e019be6-cad6-4c8e-ade1-5252d051ee2a"). InnerVolumeSpecName "kube-api-access-mxclv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:26.064192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.064150 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxclv\" (UniqueName: \"kubernetes.io/projected/6e019be6-cad6-4c8e-ade1-5252d051ee2a-kube-api-access-mxclv\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:26.447898 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.447791 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:26.703474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.703367 5988 generic.go:356] "Generic (PLEG): container finished" podID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerID="e15e3fac6a561dd3bede4f1ff980f30c720580630828b72d795aac6133ec6b58" exitCode=60 Nov 03 19:00:26.703474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.703430 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" event={"ID":"b3abfb10-50a4-49d4-b6d9-8956892d3615","Type":"ContainerDied","Data":"e15e3fac6a561dd3bede4f1ff980f30c720580630828b72d795aac6133ec6b58"} Nov 03 19:00:26.704012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.703517 5988 scope.go:122] "RemoveContainer" containerID="2981dfe78bfd8f1b2dcfc56aeb5710ec989f86d4916c9a486d9b19ff3ba3baa9" Nov 03 19:00:26.704077 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.704021 5988 scope.go:122] "RemoveContainer" containerID="e15e3fac6a561dd3bede4f1ff980f30c720580630828b72d795aac6133ec6b58" Nov 03 19:00:26.704365 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:26.704338 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz_odh-ai-gateway-infra(b3abfb10-50a4-49d4-b6d9-8956892d3615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" Nov 03 19:00:26.705818 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.705793 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" event={"ID":"6e019be6-cad6-4c8e-ade1-5252d051ee2a","Type":"ContainerDied","Data":"6d5de2dfa2758122740e16fafdb6a58b3715195395008e5425200b0dc3e478af"} Nov 03 19:00:26.705966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.705851 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk" Nov 03 19:00:26.725746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.725717 5988 scope.go:122] "RemoveContainer" containerID="f1ce151f57f6e4900bde4a57041536c5364d5443fd5903c249c44e68fd2685ee" Nov 03 19:00:26.812992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.812947 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz"] Nov 03 19:00:26.933942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.933903 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk"] Nov 03 19:00:26.941160 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:26.941123 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895540-5sdxk"] Nov 03 19:00:27.180535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:27.180497 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:27.272651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:27.272614 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" path="/var/lib/kubelet/pods/6e019be6-cad6-4c8e-ade1-5252d051ee2a/volumes" Nov 03 19:00:27.864843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:27.864817 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:27.989204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:27.989119 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3abfb10-50a4-49d4-b6d9-8956892d3615-kube-api-access-fq8pq\" (UniqueName: \"kubernetes.io/projected/b3abfb10-50a4-49d4-b6d9-8956892d3615-kube-api-access-fq8pq\") pod \"b3abfb10-50a4-49d4-b6d9-8956892d3615\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " Nov 03 19:00:27.989359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:27.989224 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3abfb10-50a4-49d4-b6d9-8956892d3615-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3abfb10-50a4-49d4-b6d9-8956892d3615-maas-api-service-ca\") pod \"b3abfb10-50a4-49d4-b6d9-8956892d3615\" (UID: \"b3abfb10-50a4-49d4-b6d9-8956892d3615\") " Nov 03 19:00:27.989945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:27.989913 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3abfb10-50a4-49d4-b6d9-8956892d3615-maas-api-service-ca" pod "b3abfb10-50a4-49d4-b6d9-8956892d3615" (UID: "b3abfb10-50a4-49d4-b6d9-8956892d3615"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:27.991400 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:27.991375 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3abfb10-50a4-49d4-b6d9-8956892d3615-kube-api-access-fq8pq" pod "b3abfb10-50a4-49d4-b6d9-8956892d3615" (UID: "b3abfb10-50a4-49d4-b6d9-8956892d3615"). InnerVolumeSpecName "kube-api-access-fq8pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:28.091403 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:28.091366 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8pq\" (UniqueName: \"kubernetes.io/projected/b3abfb10-50a4-49d4-b6d9-8956892d3615-kube-api-access-fq8pq\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:28.091403 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:28.091399 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3abfb10-50a4-49d4-b6d9-8956892d3615-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:28.721090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:28.721057 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" Nov 03 19:00:28.721275 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:28.721092 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz" event={"ID":"b3abfb10-50a4-49d4-b6d9-8956892d3615","Type":"ContainerDied","Data":"8cdc90c725850a087e0f42a4c85abc58e615212053a449960769e3924816e7d7"} Nov 03 19:00:28.721275 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:28.721158 5988 scope.go:122] "RemoveContainer" containerID="e15e3fac6a561dd3bede4f1ff980f30c720580630828b72d795aac6133ec6b58" Nov 03 19:00:28.875886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:28.875848 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz"] Nov 03 19:00:28.922278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:28.922241 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9686c3-29895540-qtvqz"] Nov 03 19:00:29.272938 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:29.272903 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" path="/var/lib/kubelet/pods/b3abfb10-50a4-49d4-b6d9-8956892d3615/volumes" Nov 03 19:00:31.721829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:31.721792 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:33.356679 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:33.356642 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:33.560176 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:33.560138 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:36.373927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:36.373881 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:40.930911 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:40.930875 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:43.661812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:43.661769 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:43.875135 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:43.875098 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:45.965907 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:45.965866 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:47.263610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:47.263579 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:47.477971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:47.477932 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:47.900791 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:47.900753 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:48.461111 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461027 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr"] Nov 03 19:00:48.461827 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461808 5988 cpu_manager.go:414] "containerMap: removing container" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.461889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461830 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.461889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461851 5988 cpu_manager.go:414] "containerMap: removing container" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.461889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461859 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.461889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461869 5988 cpu_manager.go:414] "containerMap: removing container" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.461889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461877 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.462106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461903 5988 cpu_manager.go:414] "containerMap: removing container" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.462106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461911 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.462106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461938 5988 cpu_manager.go:414] "containerMap: removing container" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.462106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.461946 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.462106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.462095 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.462266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.462107 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.462266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.462126 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.462266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.462137 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.466067 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.466042 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.555604 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2jm\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-kube-api-access-fn2jm\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.555676 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.555719 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.555824 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.555846 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b665b35c-7648-4d19-b8d3-e723680182a6-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.555898 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.555984 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b665b35c-7648-4d19-b8d3-e723680182a6-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.556058 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.556451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.556103 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.565805 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.565777 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr"] Nov 03 19:00:48.657902 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.657856 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658075 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.657920 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658075 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.657957 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2jm\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-kube-api-access-fn2jm\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658075 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658005 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658075 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658029 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658127 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658152 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b665b35c-7648-4d19-b8d3-e723680182a6-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658204 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658287 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b665b35c-7648-4d19-b8d3-e723680182a6-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658294 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658416 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658673 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.658853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.658751 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.659155 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.659137 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b665b35c-7648-4d19-b8d3-e723680182a6-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.660426 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.660401 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.660956 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.660936 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b665b35c-7648-4d19-b8d3-e723680182a6-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.715574 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.715496 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.715771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.715640 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2jm\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-kube-api-access-fn2jm\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.780874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.780836 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:48.892373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.892327 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8"] Nov 03 19:00:48.893397 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.893191 5988 cpu_manager.go:414] "containerMap: removing container" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.893397 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.893215 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.893397 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.893377 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e019be6-cad6-4c8e-ade1-5252d051ee2a" containerName="cleanup" Nov 03 19:00:48.893397 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.893396 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3abfb10-50a4-49d4-b6d9-8956892d3615" containerName="cleanup" Nov 03 19:00:48.896489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.896472 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.956980 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.956934 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8"] Nov 03 19:00:48.963228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963192 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.963507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963483 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.963649 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963622 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.963820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963697 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.963820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963725 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.963820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963812 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.964015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963906 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.964015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963927 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm4p4\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-kube-api-access-fm4p4\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:48.964015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:48.963959 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.065966 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066075 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066106 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm4p4\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-kube-api-access-fm4p4\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066139 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066228 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066290 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066320 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066360 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.066446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.066389 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.067937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.067034 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.067937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.067267 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.067937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.067493 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.067937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.067713 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.068446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.068280 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.069558 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.069529 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.070923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.070896 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.125723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.125671 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.126651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.126619 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm4p4\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-kube-api-access-fm4p4\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.213178 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.213132 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:49.221855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.221827 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr"] Nov 03 19:00:49.224812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.224785 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn"] Nov 03 19:00:49.227048 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:00:49.227019 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb665b35c_7648_4d19_b8d3_e723680182a6.slice/crio-1576740bbc0989f3f97889c585d584b681fe0dae8131a077f4567b8343402652 WatchSource:0}: Error finding container 1576740bbc0989f3f97889c585d584b681fe0dae8131a077f4567b8343402652: Status 404 returned error can't find the container with id 1576740bbc0989f3f97889c585d584b681fe0dae8131a077f4567b8343402652 Nov 03 19:00:49.229200 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.229162 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.229259 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.229235 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.229299 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.229277 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.229460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.229441 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274559 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274631 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274695 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/475f7e65-3be6-446b-9be5-86d850b81602-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274731 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274759 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/475f7e65-3be6-446b-9be5-86d850b81602-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274796 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274826 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274893 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.276704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.274996 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jq4s\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-kube-api-access-7jq4s\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.291131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.288569 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn"] Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375586 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jq4s\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-kube-api-access-7jq4s\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375656 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375713 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375749 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/475f7e65-3be6-446b-9be5-86d850b81602-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375773 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375790 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/475f7e65-3be6-446b-9be5-86d850b81602-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375823 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375839 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.375890 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.377363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.377069 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.378036 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.377472 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.378101 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.378039 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/475f7e65-3be6-446b-9be5-86d850b81602-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.378634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.378261 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.378780 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.378713 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.379210 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.378976 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.381324 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.381300 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/475f7e65-3be6-446b-9be5-86d850b81602-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.435482 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.435435 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jq4s\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-kube-api-access-7jq4s\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.435637 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.435564 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.555904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.555853 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:49.611631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.611599 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8"] Nov 03 19:00:49.640313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.639358 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.640313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.639465 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.640313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.639615 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.854378 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.854279 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" event={"ID":"b665b35c-7648-4d19-b8d3-e723680182a6","Type":"ContainerStarted","Data":"8b73a4db54a30257081cb70b2e9805ada54b61f7a4ca9af95397e3a345e4d518"} Nov 03 19:00:49.854378 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.854345 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" event={"ID":"b665b35c-7648-4d19-b8d3-e723680182a6","Type":"ContainerStarted","Data":"1576740bbc0989f3f97889c585d584b681fe0dae8131a077f4567b8343402652"} Nov 03 19:00:49.856124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.856097 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" event={"ID":"f30f3b22-0bce-490e-ba6c-d318c9c8b989","Type":"ContainerStarted","Data":"e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274"} Nov 03 19:00:49.856278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.856140 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" event={"ID":"f30f3b22-0bce-490e-ba6c-d318c9c8b989","Type":"ContainerStarted","Data":"8bc00074117221d9dacd5f0dcbd4eb65e35da8d1d784700e51aba797a7fe2382"} Nov 03 19:00:49.941466 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.941438 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn"] Nov 03 19:00:49.944973 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:00:49.944943 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475f7e65_3be6_446b_9be5_86d850b81602.slice/crio-7f8a3a021cd8214b9a81cd6188628ab1732c00b894969871bceef93eef7e4b4a WatchSource:0}: Error finding container 7f8a3a021cd8214b9a81cd6188628ab1732c00b894969871bceef93eef7e4b4a: Status 404 returned error can't find the container with id 7f8a3a021cd8214b9a81cd6188628ab1732c00b894969871bceef93eef7e4b4a Nov 03 19:00:49.948264 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.948224 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.948381 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.948332 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.948381 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.948373 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:00:49.971412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:49.971344 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podStartSLOduration=1.971324682 podStartE2EDuration="1.971324682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:49.969938766 +0000 UTC m=+1324.851354869" watchObservedRunningTime="2026-11-03 19:00:49.971324682 +0000 UTC m=+1324.852740786" Nov 03 19:00:50.091726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.088364 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podStartSLOduration=2.088332249 podStartE2EDuration="2.088332249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:50.085599188 +0000 UTC m=+1324.967015291" watchObservedRunningTime="2026-11-03 19:00:50.088332249 +0000 UTC m=+1324.969748355" Nov 03 19:00:50.213707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.213597 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:50.215075 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.215053 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:00:50.215122 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.215101 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:00:50.787288 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.786793 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:50.787288 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.786939 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:00:50.787288 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.786987 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:00:50.896908 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.896864 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" event={"ID":"475f7e65-3be6-446b-9be5-86d850b81602","Type":"ContainerStarted","Data":"043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5"} Nov 03 19:00:50.897109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:50.896936 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" event={"ID":"475f7e65-3be6-446b-9be5-86d850b81602","Type":"ContainerStarted","Data":"7f8a3a021cd8214b9a81cd6188628ab1732c00b894969871bceef93eef7e4b4a"} Nov 03 19:00:51.036235 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.033231 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:51.036235 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.035001 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podStartSLOduration=2.034979749 podStartE2EDuration="2.034979749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:51.031474458 +0000 UTC m=+1325.912890577" watchObservedRunningTime="2026-11-03 19:00:51.034979749 +0000 UTC m=+1325.916395890" Nov 03 19:00:51.217871 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.217130 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:00:51.217871 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.217193 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:00:51.564781 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.564126 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:51.573857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.573817 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:00:51.574138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.574105 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:00:51.784709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.781609 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:00:51.784709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:51.781670 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:00:52.214376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:52.214333 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:00:52.215053 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:52.214997 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:00:52.559768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:52.558162 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:00:52.559768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:52.558231 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:00:52.789709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:52.789182 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:00:52.789709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:52.789249 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:00:53.214491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:53.214451 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:00:53.215621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:53.215584 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:00:53.560776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:53.560139 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:00:53.560776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:53.560211 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:00:53.791719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:53.791130 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:00:53.791719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:53.791196 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:00:54.225453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:54.222804 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:00:54.225453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:54.222873 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:00:54.565770 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:54.563130 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:00:54.565770 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:54.563198 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:00:54.785712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:54.785466 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:00:54.785712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:54.785532 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:00:55.215709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:55.215107 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:00:55.215709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:55.215194 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:00:55.570706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:55.567146 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:00:55.570706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:55.567229 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:00:55.783706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:55.783120 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 19:00:55.783706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:55.783182 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 19:00:55.939744 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:00:55.939643 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:00:56.214153 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:56.214067 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 19:00:56.214153 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:56.214138 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 19:00:56.557757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:56.557046 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:00:56.557757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:56.557123 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:00:56.567987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:56.567930 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:00:57.200706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.200473 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:57.201223 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.200967 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:57.203833 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.202387 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:00:57.555596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.554234 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:57.555815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.555793 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:57.556744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.556644 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:00:57.873231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.873198 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:57.873954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.873924 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:00:57.874899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:00:57.874874 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:01:01.307460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:01.307414 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:03.992063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:03.992030 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh"] Nov 03 19:01:03.996137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:03.996114 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096311 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096378 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096445 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb291fbd-f786-4780-8fff-e07e380f99b9-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096467 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096530 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096552 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096571 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096595 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njt9k\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-kube-api-access-njt9k\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.096854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.096640 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.099217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.099187 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh"] Nov 03 19:01:04.198349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198310 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198361 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198424 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb291fbd-f786-4780-8fff-e07e380f99b9-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198449 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198503 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198539 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198566 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198600 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njt9k\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-kube-api-access-njt9k\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198666 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.198982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.198949 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.199169 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.199145 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-certs\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.199288 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.199263 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-credential-socket\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.199383 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.199289 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-data\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.199473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.199457 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb291fbd-f786-4780-8fff-e07e380f99b9-istiod-ca-cert\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.200882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.200859 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-envoy\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.200963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.200895 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-podinfo\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.256352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.256284 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-token\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.256479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.256426 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njt9k\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-kube-api-access-njt9k\") pod \"e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.309695 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.309648 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:04.730435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.730406 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh"] Nov 03 19:01:04.732548 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:01:04.732514 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb291fbd_f786_4780_8fff_e07e380f99b9.slice/crio-a47e1d6ea5a2603f22a26f59ee7bfec3a6a56f0b5366f382078625e27e19884b WatchSource:0}: Error finding container a47e1d6ea5a2603f22a26f59ee7bfec3a6a56f0b5366f382078625e27e19884b: Status 404 returned error can't find the container with id a47e1d6ea5a2603f22a26f59ee7bfec3a6a56f0b5366f382078625e27e19884b Nov 03 19:01:04.735328 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.735295 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:04.735421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.735357 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:04.735421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:04.735386 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:05.052816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:05.052726 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" event={"ID":"cb291fbd-f786-4780-8fff-e07e380f99b9","Type":"ContainerStarted","Data":"e8fe6c2b6bb27eea35b100ad25ae99c15a721c0a871fb7dc8e1c2e9f8f5e6d7a"} Nov 03 19:01:05.052816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:05.052795 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" event={"ID":"cb291fbd-f786-4780-8fff-e07e380f99b9","Type":"ContainerStarted","Data":"a47e1d6ea5a2603f22a26f59ee7bfec3a6a56f0b5366f382078625e27e19884b"} Nov 03 19:01:05.166902 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:05.166844 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podStartSLOduration=2.166826705 podStartE2EDuration="2.166826705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:05.166267646 +0000 UTC m=+1340.047683760" watchObservedRunningTime="2026-11-03 19:01:05.166826705 +0000 UTC m=+1340.048242809" Nov 03 19:01:05.310107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:05.310025 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:05.311867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:05.311840 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:01:05.311997 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:05.311888 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:01:05.929308 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:05.929264 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:06.310342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:06.310250 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:01:06.310342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:06.310313 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:01:07.310210 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:07.310174 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:01:07.310416 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:07.310238 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:01:08.310618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:08.310584 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:01:08.311149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:08.310640 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:01:09.310761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:09.310726 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:01:09.311130 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:09.310784 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:01:10.310399 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:10.310357 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:01:10.310554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:10.310426 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:01:10.478497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:10.478451 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:11.310847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:11.310813 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 19:01:11.311024 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:11.310864 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 19:01:12.315496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:12.315465 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:12.316017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:12.315732 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:12.316425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:12.316408 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:01:15.121723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:15.121660 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:19.714883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:19.714847 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:20.064146 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.064073 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9"] Nov 03 19:01:20.069339 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.069320 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.108818 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.108791 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.108998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.108822 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.108998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.108841 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xj6p\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-kube-api-access-6xj6p\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.108998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.108951 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.108998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.108995 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.109241 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.109058 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.109241 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.109153 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.109241 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.109188 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.109408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.109298 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.167217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.167177 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9"] Nov 03 19:01:20.210535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210500 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210552 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210573 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210628 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210796 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210817 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210834 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xj6p\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-kube-api-access-6xj6p\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210867 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.210919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.210909 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.211167 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.211052 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.211167 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.211147 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.211281 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.211259 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.211336 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.211281 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.211657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.211638 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.213112 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.213090 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.213321 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.213304 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.267927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.267903 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.268353 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.268330 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xj6p\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-kube-api-access-6xj6p\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.379662 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.379633 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99"] Nov 03 19:01:20.383460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.383425 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:20.385405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.385391 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.456746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.456717 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99"] Nov 03 19:01:20.515056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515020 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43e5825-60f0-4768-be89-bd2136c65a57-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515201 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515084 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515209 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515314 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515297 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcw6\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-kube-api-access-ppcw6\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515387 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515467 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515539 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515525 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43e5825-60f0-4768-be89-bd2136c65a57-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515625 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.515776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.515758 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617144 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617119 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617174 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcw6\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-kube-api-access-ppcw6\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617206 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617236 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617256 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43e5825-60f0-4768-be89-bd2136c65a57-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617274 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:01:20.617591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617324 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617394 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617429 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43e5825-60f0-4768-be89-bd2136c65a57-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617466 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617558 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-credential-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617861 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617634 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-socket\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617861 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617748 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-data\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.617929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.617859 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-certs\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.618104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.618083 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43e5825-60f0-4768-be89-bd2136c65a57-istiod-ca-cert\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.619771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.619734 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-envoy\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.619969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.619952 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43e5825-60f0-4768-be89-bd2136c65a57-istio-podinfo\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.674483 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.674415 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcw6\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-kube-api-access-ppcw6\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.675067 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.675047 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-istio-token\") pod \"e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.713992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.713967 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:20.807202 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.807172 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9"] Nov 03 19:01:20.812043 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:01:20.812015 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11ce5f1_b784_41dd_a73d_3e5f5d196dbd.slice/crio-d2fdc76e2504a94999e5a60a3389bf24c483d501b375b48292987d45e4631412 WatchSource:0}: Error finding container d2fdc76e2504a94999e5a60a3389bf24c483d501b375b48292987d45e4631412: Status 404 returned error can't find the container with id d2fdc76e2504a94999e5a60a3389bf24c483d501b375b48292987d45e4631412 Nov 03 19:01:20.814901 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.814862 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:20.814976 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.814949 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:20.815022 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:20.814991 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:21.086589 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.086556 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99"] Nov 03 19:01:21.087295 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:01:21.087257 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43e5825_60f0_4768_be89_bd2136c65a57.slice/crio-24eec94535732dd03754cea81b174d7e8548eb088395d906ee50e828c1932896 WatchSource:0}: Error finding container 24eec94535732dd03754cea81b174d7e8548eb088395d906ee50e828c1932896: Status 404 returned error can't find the container with id 24eec94535732dd03754cea81b174d7e8548eb088395d906ee50e828c1932896 Nov 03 19:01:21.089954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.089920 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:21.090051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.089995 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:21.090051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.090028 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:21.170184 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.170148 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" event={"ID":"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd","Type":"ContainerStarted","Data":"73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630"} Nov 03 19:01:21.170347 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.170224 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" event={"ID":"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd","Type":"ContainerStarted","Data":"d2fdc76e2504a94999e5a60a3389bf24c483d501b375b48292987d45e4631412"} Nov 03 19:01:21.172117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.172085 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" event={"ID":"a43e5825-60f0-4768-be89-bd2136c65a57","Type":"ContainerStarted","Data":"24eec94535732dd03754cea81b174d7e8548eb088395d906ee50e828c1932896"} Nov 03 19:01:21.287697 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.287639 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podStartSLOduration=1.287625795 podStartE2EDuration="1.287625795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:21.284248158 +0000 UTC m=+1356.165664260" watchObservedRunningTime="2026-11-03 19:01:21.287625795 +0000 UTC m=+1356.169041896" Nov 03 19:01:21.384893 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.384540 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:21.385880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.385853 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:01:21.386023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:21.385912 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:01:22.207705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:22.207543 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" event={"ID":"a43e5825-60f0-4768-be89-bd2136c65a57","Type":"ContainerStarted","Data":"d5610ecb38181177136e9185de4195842a43f33aa247a5b0728bba5cfabe2f2e"} Nov 03 19:01:22.390708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:22.389133 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:01:22.390708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:22.389224 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:01:22.728788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:22.728750 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:22.729713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:22.729282 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:01:22.729713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:22.729336 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 19:01:23.386707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.386102 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:01:23.386707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.386168 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:01:23.737756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.737565 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:01:23.737756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.737631 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 19:01:23.903893 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.903783 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:23.934891 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.934836 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:23.935801 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.935763 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:23.949858 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.949815 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:23.981880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:23.981828 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:24.097601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:24.094965 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 19:01:24.387761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:24.387317 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:01:24.387761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:24.387377 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:01:24.714593 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:24.714493 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:01:24.714593 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:24.714556 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 19:01:25.403267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:25.402539 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:01:25.403267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:25.402605 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:01:25.714478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:25.714391 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:01:25.714478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:25.714449 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 19:01:26.000524 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:01:26.000430 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:01:26.385719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:26.385416 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:01:26.385719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:26.385477 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:01:26.714603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:26.714514 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:01:26.714603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:26.714578 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 19:01:26.976558 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:26.976466 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:27.191252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:27.191217 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:27.384821 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:27.384783 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 19:01:27.384983 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:27.384845 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 19:01:27.714868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:27.714769 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 19:01:27.714868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:27.714826 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 19:01:28.633994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:28.633937 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:28.634870 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:28.634844 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:28.635895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:28.635872 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:01:28.796432 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:28.796360 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podStartSLOduration=8.796340501 podStartE2EDuration="8.796340501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:22.32538729 +0000 UTC m=+1357.206803423" watchObservedRunningTime="2026-11-03 19:01:28.796340501 +0000 UTC m=+1363.677756604" Nov 03 19:01:29.003857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:29.003767 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:29.004382 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:29.004360 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:29.005414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:29.005389 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:01:29.800525 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:29.800495 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:34.519672 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:34.519634 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:39.237532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:39.237480 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:43.912698 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:43.912656 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:48.370331 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:48.370294 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:48.581286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:48.581252 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:50.033207 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.033162 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4"] Nov 03 19:01:50.040906 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.040879 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.091967 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.091927 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4"] Nov 03 19:01:50.134740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.134711 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9686c3-openshift-default-dockercfg-kfklz\"" Nov 03 19:01:50.185642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.185601 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.185857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.185667 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.185857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.185837 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.185976 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.185879 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.185976 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.185905 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.186079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.185998 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.186152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.186137 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.186209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.186178 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj6lw\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-kube-api-access-rj6lw\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.186306 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.186294 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287135 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287211 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287248 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287275 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287300 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287342 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287458 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287834 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287504 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj6lw\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-kube-api-access-rj6lw\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.287834 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287587 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.288166 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.287974 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-certs\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.288166 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.288061 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-credential-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.288166 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.288112 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-data\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.288477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.288227 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-socket\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.288477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.288289 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istiod-ca-cert\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.289582 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.289561 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-envoy\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.290394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.290374 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-podinfo\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.344366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.344336 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-token\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.344714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.344669 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj6lw\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-kube-api-access-rj6lw\") pod \"e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.354999 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.354972 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:50.777161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.777128 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4"] Nov 03 19:01:50.779024 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:01:50.778991 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8de12aa_96f8_4e54_ad0d_6e5fa89202ce.slice/crio-5bacf69ea61ab5383fef1f764d7e4167127d60a34a38bdbb6ddc9f1e85e6db5b WatchSource:0}: Error finding container 5bacf69ea61ab5383fef1f764d7e4167127d60a34a38bdbb6ddc9f1e85e6db5b: Status 404 returned error can't find the container with id 5bacf69ea61ab5383fef1f764d7e4167127d60a34a38bdbb6ddc9f1e85e6db5b Nov 03 19:01:50.781890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.781849 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:50.781992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.781939 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:50.781992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:50.781986 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:01:51.469629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:51.469593 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" event={"ID":"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce","Type":"ContainerStarted","Data":"e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4"} Nov 03 19:01:51.470043 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:51.469659 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" event={"ID":"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce","Type":"ContainerStarted","Data":"5bacf69ea61ab5383fef1f764d7e4167127d60a34a38bdbb6ddc9f1e85e6db5b"} Nov 03 19:01:51.638368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:51.638317 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podStartSLOduration=2.638303203 podStartE2EDuration="2.638303203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:51.636486038 +0000 UTC m=+1386.517902140" watchObservedRunningTime="2026-11-03 19:01:51.638303203 +0000 UTC m=+1386.519719304" Nov 03 19:01:52.371706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:52.368252 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:01:52.372029 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:52.372004 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:01:52.372205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:52.372180 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:01:53.371161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:53.371117 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:01:53.371621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:53.371223 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:01:53.490696 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:53.490644 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:54.368706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:54.364844 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:01:54.368706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:54.364959 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:01:55.372530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:55.372490 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:01:55.373077 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:55.372561 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:01:55.941255 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:01:55.941207 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:01:56.370712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:56.367080 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:01:56.370712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:56.367164 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:01:57.356038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:57.356001 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:01:57.356467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:57.356067 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:01:58.458341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:58.458302 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:01:59.418919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:59.418881 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:01:59.419172 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:59.419142 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:59.946134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:59.946091 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:01:59.947727 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:01:59.947695 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.362864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.362782 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.444568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.444526 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:02:00.466131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.465767 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:00.466131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.465822 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.868813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.868762 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.903236 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.903183 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 19:02:00.952994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.952950 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.992025 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.991986 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:00.992421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.956867 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.054712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:00.994766 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:02:01.092709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.090826 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.113708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.108063 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.115750 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.115712 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.258373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.258290 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:02:01.299810 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.267984 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:02:01.378910 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.378859 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.510765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.447640 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.590239 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.572831 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:02:01.590580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.590561 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:02:01.590794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.590777 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 19:02:01.599716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.583219 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.921827 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.921775 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.922481 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.922456 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:02:01.950841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.950792 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:01.951455 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:01.951422 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:02:02.034709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.027215 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 19:02:02.058707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.036967 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.060542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.058977 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 19:02:02.061485 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.061188 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 19:02:02.062061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.062044 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 19:02:02.095965 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.095928 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.096556 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.096532 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 19:02:02.238144 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.238063 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:02:02.389433 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.389383 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Nov 03 19:02:02.635030 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.634918 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:02.649121 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.649087 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:02.649457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.649435 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 03 19:02:02.649961 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.649941 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 19:02:02.825959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.815260 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:02:02.857427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.857398 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:02:02.865768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:02.865741 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 19:02:03.068854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.068817 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:03.095707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.070204 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:02:03.105707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.096959 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:03.119713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.119671 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 19:02:03.149779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.149752 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 19:02:03.169419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.169392 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 19:02:03.208759 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.208292 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 19:02:03.254023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.250995 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:03.619535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.619501 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:03.842537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:03.842047 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:04.628614 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:04.628576 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:06.599853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:06.599819 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:06.818076 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:06.818048 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:07.027289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:07.027220 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:07.246250 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:07.246206 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:09.920509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:09.920457 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:10.138359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:10.138317 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:11.371661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:11.371626 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:11.588516 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:11.588475 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:11.799290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:11.799189 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:12.400671 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:12.400620 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:12.638341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:12.638299 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:12.943058 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:12.943024 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:14.527125 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:14.527084 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:15.153378 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:15.153272 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:15.425017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:15.424914 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:16.579469 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:16.579427 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:16.831678 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:16.831592 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:17.833237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:17.833193 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:19.197694 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:19.197651 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:19.509380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:19.509301 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:19.867337 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:19.867306 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:21.071944 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.071910 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:21.287826 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.287796 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:21.574885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.574860 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4"] Nov 03 19:02:21.586647 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.586611 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:21.686372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.686340 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6f198bfb\"" Nov 03 19:02:21.690932 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.690892 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6f198bfb-dockercfg-67mvw\"" Nov 03 19:02:21.692467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.692440 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4"] Nov 03 19:02:21.728997 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.728961 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:21.729331 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.729311 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqbp\" (UniqueName: \"kubernetes.io/projected/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-kube-api-access-6fqbp\") pod \"payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:21.830676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.830589 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqbp\" (UniqueName: \"kubernetes.io/projected/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-kube-api-access-6fqbp\") pod \"payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:21.830862 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.830741 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:21.831468 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.831449 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-plugins-config-volume\") pod \"payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:21.889380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.889345 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqbp\" (UniqueName: \"kubernetes.io/projected/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-kube-api-access-6fqbp\") pod \"payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:21.901209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:21.901184 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:22.273239 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:22.273206 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4"] Nov 03 19:02:22.276532 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:02:22.276504 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcb07b3_c2a2_4074_b1e5_ed4c73121c15.slice/crio-0639dfa85ab32fdc201122864203e92e297e9cc48f3d9ceca7f036e96162a006 WatchSource:0}: Error finding container 0639dfa85ab32fdc201122864203e92e297e9cc48f3d9ceca7f036e96162a006: Status 404 returned error can't find the container with id 0639dfa85ab32fdc201122864203e92e297e9cc48f3d9ceca7f036e96162a006 Nov 03 19:02:22.966800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:22.966765 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" event={"ID":"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15","Type":"ContainerStarted","Data":"c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7"} Nov 03 19:02:22.966984 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:22.966831 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" event={"ID":"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15","Type":"ContainerStarted","Data":"0639dfa85ab32fdc201122864203e92e297e9cc48f3d9ceca7f036e96162a006"} Nov 03 19:02:22.967033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:22.966981 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:23.029342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:23.029295 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:23.077909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:23.077857 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" podStartSLOduration=2.07784181 podStartE2EDuration="2.07784181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:23.076508268 +0000 UTC m=+1417.957924370" watchObservedRunningTime="2026-11-03 19:02:23.07784181 +0000 UTC m=+1417.959257912" Nov 03 19:02:23.870782 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:23.870740 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:24.098587 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:24.098554 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:24.463963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:24.463876 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:24.686436 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:24.686390 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:24.932797 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:24.932768 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:25.936703 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:02:25.936655 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:02:27.853799 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:27.853758 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:32.422099 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:32.422056 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:33.885676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:33.885642 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:33.990607 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:33.990576 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:02:34.091145 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.091109 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr"] Nov 03 19:02:34.094676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.094647 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.120268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.120229 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:34.196787 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.196706 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr"] Nov 03 19:02:34.276783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.276755 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq"] Nov 03 19:02:34.278914 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.278874 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.278989 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.279045 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4tq\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-kube-api-access-wz4tq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279156 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.279097 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.279211 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279288 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.279253 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279382 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.279357 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.279397 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.279542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.279509 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.280577 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.280560 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.349376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.349343 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq"] Nov 03 19:02:34.380471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380437 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.380600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380482 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.380600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380499 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.380600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380581 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.380740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380626 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.380740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380657 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.380881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380856 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfshd\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-kube-api-access-xfshd\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.381012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380906 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.381012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380942 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380967 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.380971 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381011 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381021 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.381222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381059 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d618dc2-5662-46bb-b1b6-9824a6e01924-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.381222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381075 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.381222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381211 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381240 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.381408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381270 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.381408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381312 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381384 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4tq\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-kube-api-access-wz4tq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381447 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.381629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.381590 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.382106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.382084 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.383331 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.383304 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.383491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.383473 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.440839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.440773 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.441254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.441233 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4tq\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-kube-api-access-wz4tq\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.482631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482563 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.482631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482606 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.482895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482721 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.482895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482760 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.482895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482807 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfshd\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-kube-api-access-xfshd\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.482895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482839 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.482895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482884 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.483152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482919 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d618dc2-5662-46bb-b1b6-9824a6e01924-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.483152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.482990 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.483152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.483078 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-credential-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.483308 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.483167 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-socket\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.483479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.483460 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-certs\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.483555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.483486 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-data\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.483555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.483544 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d618dc2-5662-46bb-b1b6-9824a6e01924-istiod-ca-cert\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.484888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.484864 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-envoy\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.485253 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.485231 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-podinfo\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.549227 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.549200 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfshd\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-kube-api-access-xfshd\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.549623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.549604 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-token\") pod \"e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.562340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.562318 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:34.593817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.593793 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:34.708032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.707829 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:34.766290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.766220 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:34.997366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:34.997332 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:35.016655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.016590 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq"] Nov 03 19:02:35.019926 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:02:35.019899 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d618dc2_5662_46bb_b1b6_9824a6e01924.slice/crio-7da98153ed2076dc1db8aed269ea6b4ce0ae7b69d1a95ad1158feebbbc60a705 WatchSource:0}: Error finding container 7da98153ed2076dc1db8aed269ea6b4ce0ae7b69d1a95ad1158feebbbc60a705: Status 404 returned error can't find the container with id 7da98153ed2076dc1db8aed269ea6b4ce0ae7b69d1a95ad1158feebbbc60a705 Nov 03 19:02:35.022394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.022363 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.022476 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.022425 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.022476 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.022455 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.053599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.053574 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" event={"ID":"3d618dc2-5662-46bb-b1b6-9824a6e01924","Type":"ContainerStarted","Data":"7da98153ed2076dc1db8aed269ea6b4ce0ae7b69d1a95ad1158feebbbc60a705"} Nov 03 19:02:35.092388 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.092362 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp"] Nov 03 19:02:35.096406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.096388 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.130794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.130760 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr"] Nov 03 19:02:35.143595 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.143556 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.143720 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.143639 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.143720 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.143698 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.155900 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.155879 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp"] Nov 03 19:02:35.193036 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193009 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193168 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193066 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54t8s\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-kube-api-access-54t8s\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193168 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193103 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193168 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193134 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193327 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193294 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54416f7b-5912-4ae0-a003-4cd0addb5113-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193403 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193383 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193432 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193459 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.193580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.193550 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.251059 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.251029 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:35.294527 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294437 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.294527 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294488 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54t8s\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-kube-api-access-54t8s\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.294527 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294518 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.294868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294546 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.294868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294665 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54416f7b-5912-4ae0-a003-4cd0addb5113-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.294868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294764 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.294868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294802 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.294868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294827 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.295136 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.294908 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.295395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.295367 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-credential-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.296095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.296056 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-data\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.296966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.296939 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54416f7b-5912-4ae0-a003-4cd0addb5113-istiod-ca-cert\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.297082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.297068 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-certs\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.297353 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.297331 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-socket\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.297695 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.297663 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-envoy\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.299563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.299539 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-podinfo\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.361460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.361426 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-token\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.362873 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.362843 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54t8s\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-kube-api-access-54t8s\") pod \"e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.412293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.412264 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:35.458166 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.458132 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:35.617763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.617734 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9"] Nov 03 19:02:35.622701 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.622659 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.683372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.683341 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9"] Nov 03 19:02:35.808503 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808452 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.808754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808567 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.808754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808749 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.808883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808777 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vl88\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-kube-api-access-9vl88\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.808883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808849 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.809025 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808890 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.809025 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808921 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.809025 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.808945 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.809169 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.809039 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.856250 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.856216 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp"] Nov 03 19:02:35.892343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.892305 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.892559 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.892536 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.892926 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.892903 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:35.910060 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.910036 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.910435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.910414 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-credential-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.910572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.910554 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.910776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.910759 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.911350 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.911102 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-socket\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.911453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.911128 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istiod-ca-cert\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.911793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.911774 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.912147 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.912119 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vl88\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-kube-api-access-9vl88\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.912890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.912873 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.914238 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.912071 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-certs\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.914442 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.914183 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.914590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.914574 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.915631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.914971 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.915945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.914863 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-data\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.926340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.926296 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-envoy\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:35.933371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:35.933323 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-podinfo\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:36.000878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.000807 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vl88\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-kube-api-access-9vl88\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:36.002215 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.002160 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-token\") pod \"e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:36.096383 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.096009 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" event={"ID":"54416f7b-5912-4ae0-a003-4cd0addb5113","Type":"ContainerStarted","Data":"95050668c994e888a0fa6bfded3729c42670a8109390b952b0e0db1b0a874c40"} Nov 03 19:02:36.100877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.098460 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" event={"ID":"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d","Type":"ContainerStarted","Data":"dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1"} Nov 03 19:02:36.100877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.098511 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" event={"ID":"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d","Type":"ContainerStarted","Data":"20a29693ce1ec27f135b581a8546e5aa88148b906b493f43b4d5182c95c5fc10"} Nov 03 19:02:36.103530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.103461 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" event={"ID":"3d618dc2-5662-46bb-b1b6-9824a6e01924","Type":"ContainerStarted","Data":"0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f"} Nov 03 19:02:36.222487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.222386 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podStartSLOduration=2.22237393 podStartE2EDuration="2.22237393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:36.219933936 +0000 UTC m=+1431.101350072" watchObservedRunningTime="2026-11-03 19:02:36.22237393 +0000 UTC m=+1431.103790031" Nov 03 19:02:36.240073 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.240049 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:36.335706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.335356 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podStartSLOduration=2.335341276 podStartE2EDuration="2.335341276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:36.33451237 +0000 UTC m=+1431.215928475" watchObservedRunningTime="2026-11-03 19:02:36.335341276 +0000 UTC m=+1431.216757377" Nov 03 19:02:36.609709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.608800 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:36.610444 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.610087 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:36.610444 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.610143 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:36.701714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.701378 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9"] Nov 03 19:02:36.708903 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.708870 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:36.710364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.710339 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:36.710477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.710381 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:36.722148 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:02:36.722118 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023d7eaf_9220_48e0_ab41_3a09a7de82e3.slice/crio-3e94197ae0bd7054a67e846c798933e1760f1381c5821d46250a9b62e29c1dda WatchSource:0}: Error finding container 3e94197ae0bd7054a67e846c798933e1760f1381c5821d46250a9b62e29c1dda: Status 404 returned error can't find the container with id 3e94197ae0bd7054a67e846c798933e1760f1381c5821d46250a9b62e29c1dda Nov 03 19:02:36.724998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.724963 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:36.725112 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.725048 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:36.725112 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:36.725099 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:37.139933 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.139070 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" event={"ID":"54416f7b-5912-4ae0-a003-4cd0addb5113","Type":"ContainerStarted","Data":"b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737"} Nov 03 19:02:37.142250 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.142221 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" event={"ID":"023d7eaf-9220-48e0-ab41-3a09a7de82e3","Type":"ContainerStarted","Data":"75725ac282a1982e8b67b85ae7009036f15f0fad15cd646ebbe49c58b18dad68"} Nov 03 19:02:37.142379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.142274 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" event={"ID":"023d7eaf-9220-48e0-ab41-3a09a7de82e3","Type":"ContainerStarted","Data":"3e94197ae0bd7054a67e846c798933e1760f1381c5821d46250a9b62e29c1dda"} Nov 03 19:02:37.240537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.240506 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:37.241627 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.241598 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.241824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.241645 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:37.406762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.401367 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podStartSLOduration=2.401344298 podStartE2EDuration="2.401344298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:37.271961665 +0000 UTC m=+1432.153377768" watchObservedRunningTime="2026-11-03 19:02:37.401344298 +0000 UTC m=+1432.282760401" Nov 03 19:02:37.419705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.416164 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:37.419705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.416306 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.419705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.416351 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:37.610717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.601781 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.610717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.601846 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:37.644710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.636829 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podStartSLOduration=2.636808851 podStartE2EDuration="2.636808851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:37.405065189 +0000 UTC m=+1432.286481304" watchObservedRunningTime="2026-11-03 19:02:37.636808851 +0000 UTC m=+1432.518224953" Nov 03 19:02:37.773769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.771676 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.773769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.771759 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:37.830623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.830585 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:02:37.944618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:37.944555 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:38.310710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.302176 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.310710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.302233 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:38.417709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.417266 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.417709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.417324 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:38.601732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.596994 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.601732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.597050 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:38.718782 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.712377 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.718782 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:38.712435 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:39.060765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.043951 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.085710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.082767 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.136709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.135562 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.267721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.263799 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.288128 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.288096 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.288416 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.288386 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:39.434713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.394948 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.434713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.412570 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.434713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.412619 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:39.594937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.594821 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.594937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.594878 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:39.655721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.647652 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.789767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.729941 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.853722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.826785 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.856268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.856215 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:39.909729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.883358 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.937724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.911180 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-247bb" podUID="79aabaf9-34bb-44c0-a979-d83452d3b6a8" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:39.937724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:39.911287 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.017834 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.016333 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.046657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.046567 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.047341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.047319 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:02:40.249818 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.134217 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 19:02:40.251542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.136187 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.300761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.288772 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 03 19:02:40.393712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.381857 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 19:02:40.393712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.381933 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 19:02:40.393712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.381995 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 19:02:40.393712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.384709 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 03 19:02:40.548707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.539654 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.548707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.539750 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:40.548707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.540291 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.548707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.540331 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:40.654447 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.653865 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.655125 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.654754 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:02:40.673557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.671774 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.673557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.672101 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 19:02:40.680716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.680208 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.680716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.680258 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:40.765804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.765643 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.765804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.765738 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:40.833821 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.833772 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.859711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.835802 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:02:40.906063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.905964 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:40.907428 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.907403 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 19:02:40.933026 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:40.932981 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.031587 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.031554 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:02:41.058708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.056797 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.134709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.074771 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.172815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.172710 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.298733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.256773 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.298733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.257988 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 19:02:41.298733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.292347 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.661730 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.643381 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:41.661730 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.643449 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:41.703732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.677924 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:41.703732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.677983 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:41.715040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.714998 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:41.811710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.768196 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:42.195707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.910100 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.206854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.206812 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:42.216784 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.944907 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 19:02:42.232767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.944945 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.236749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.232986 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:42.237829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.094664 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:42.238335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:41.962118 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:42.245836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.045748 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:02:42.259798 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.049130 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 19:02:42.271522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.271482 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 19:02:42.307829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.286306 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 19:02:42.331714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.295749 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 03 19:02:42.382975 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.346900 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.425708 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:02:42.400240 5988 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.115s" Nov 03 19:02:42.489773 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.443804 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.631707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.629539 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:42.638708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.634757 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:42.663768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.661918 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.664336 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.664302 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:42.704164 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.704121 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:42.708382 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.708335 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.708637 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.708601 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:42.716954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:42.716924 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.077034 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.076986 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:43.238893 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.238853 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.242372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.239134 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 19:02:43.255661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.239162 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 19:02:43.256940 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.242029 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:43.257580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.257551 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:43.278896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.278860 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 19:02:43.304706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.279407 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Nov 03 19:02:43.329757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.321764 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:43.329757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.322158 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 19:02:43.329757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.323134 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.376707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.359976 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.377451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.377415 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.449711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.441913 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:43.463533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.463480 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:43.633904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.633814 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:43.634141 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.634115 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:43.737901 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.737854 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:43.800493 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.800463 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:02:43.854708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.808992 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:43.855652 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:43.855614 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:44.230394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.230323 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Nov 03 19:02:44.278910 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.278868 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.280122 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.280090 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.307031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.306987 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.315705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.308348 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.315988 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.315954 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:44.327127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.327090 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.441541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.441487 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 03 19:02:44.473744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.472786 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.473744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.473238 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Nov 03 19:02:44.518344 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.518249 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.522803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.522767 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:44.612708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.601704 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.612708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.601760 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:44.718708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.718228 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:44.718708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.718286 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:44.740707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.739776 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.806756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.804521 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:44.838103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:44.838047 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:02:45.156705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.155769 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.242486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.242451 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.242741 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.242714 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:45.323445 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.323405 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 19:02:45.341230 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.341190 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.379758 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.375220 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 19:02:45.439013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.438869 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:02:45.539478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.465525 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.539478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.465607 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:45.539478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.504965 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.539478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.506914 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 19:02:45.539478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.537388 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.620345 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.620296 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.623813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.623775 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 19:02:45.737551 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.737470 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:45.737841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.737812 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:45.775707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.753217 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:45.802929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.802899 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 19:02:45.836225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:45.836187 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 19:02:46.240632 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.240578 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.241038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.241004 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:46.243791 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.243758 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:46.244855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.244832 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:46.437006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.436962 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Nov 03 19:02:46.443343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.443308 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 19:02:46.510356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.510271 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:02:46.511171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.511133 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.511320 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.511195 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:46.511880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.511848 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:46.512934 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.512913 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 19:02:46.610336 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.609491 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 19:02:46.709751 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.709586 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 19:02:46.709751 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:46.709645 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 19:02:47.244558 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.244369 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 19:02:47.244558 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.244427 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 19:02:47.413707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.412907 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 19:02:47.413707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.412966 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 19:02:47.428987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.427361 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:47.428987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.428832 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:47.431880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.431856 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:02:47.560807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.560390 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:47.779707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:47.779311 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:48.528706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:48.528227 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:48.630708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:48.627096 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:48.630708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:48.628189 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:48.632122 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:48.632097 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:02:48.825763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:48.808779 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:48.825763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:48.808845 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:48.835754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:48.828328 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:49.175845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.175772 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:49.176856 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.176830 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:49.177953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.177930 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:02:49.200514 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.200483 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg"] Nov 03 19:02:49.205941 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.205917 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.245708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.245322 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:49.245708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.245397 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:49.269601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.269560 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.269844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.269645 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.273927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.273889 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cnnv\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-kube-api-access-7cnnv\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.274092 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.273947 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.274092 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.273991 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.274092 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.274050 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.274341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.274111 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.274341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.274156 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.274341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.274301 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.288472 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.288434 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg"] Nov 03 19:02:49.376008 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.375955 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376008 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376007 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376076 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376103 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376150 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376294 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cnnv\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-kube-api-access-7cnnv\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376322 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376342 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376399 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.376976 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376934 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-data\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.377104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.376989 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-certs\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.377293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.377264 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.377802 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.377670 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-credential-socket\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.378335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.378299 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istiod-ca-cert\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.380021 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.379993 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-envoy\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.380884 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.380859 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-podinfo\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.444304 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.444232 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-token\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.444479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.444447 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cnnv\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-kube-api-access-7cnnv\") pod \"e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.523218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.523183 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:49.704041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:49.703788 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:50.010240 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.010210 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg"] Nov 03 19:02:50.010872 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:02:50.010847 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f671f0_b432_4cb0_8c2c_d61d5d0d5da7.slice/crio-4ca89713e902626ea968dd6ccbfece35006b21ff6dd620e9d2e145e33f315889 WatchSource:0}: Error finding container 4ca89713e902626ea968dd6ccbfece35006b21ff6dd620e9d2e145e33f315889: Status 404 returned error can't find the container with id 4ca89713e902626ea968dd6ccbfece35006b21ff6dd620e9d2e145e33f315889 Nov 03 19:02:50.014003 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.013973 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:50.014084 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.014046 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:50.014084 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.014077 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:02:50.286909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.286816 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:02:50.286909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.286879 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:02:50.492542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.492509 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:50.493391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.493354 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:50.494196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.494178 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:02:50.573336 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.573295 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" event={"ID":"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7","Type":"ContainerStarted","Data":"17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39"} Nov 03 19:02:50.573545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.573361 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" event={"ID":"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7","Type":"ContainerStarted","Data":"4ca89713e902626ea968dd6ccbfece35006b21ff6dd620e9d2e145e33f315889"} Nov 03 19:02:50.839934 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:50.839813 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podStartSLOduration=1.839791248 podStartE2EDuration="1.839791248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:50.837413271 +0000 UTC m=+1445.718829385" watchObservedRunningTime="2026-11-03 19:02:50.839791248 +0000 UTC m=+1445.721207353" Nov 03 19:02:51.035557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.035519 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:51.524062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.524023 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:51.525368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.525341 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:02:51.525490 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.525392 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:02:51.666378 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.666330 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:51.874768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.874715 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6"] Nov 03 19:02:51.879227 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.879205 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:51.927777 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.927742 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntx2w\" (UniqueName: \"kubernetes.io/projected/39b0c961-1859-4198-922c-5ff1780200a9-kube-api-access-ntx2w\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:51.927978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.927879 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:51.927978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.927915 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0c961-1859-4198-922c-5ff1780200a9-openshift-service-ca\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:51.928099 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.928085 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-metrics-certs\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:51.928344 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.928330 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-90774e61-serving-cert\"" Nov 03 19:02:51.935148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:51.935124 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6"] Nov 03 19:02:52.029915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.029880 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntx2w\" (UniqueName: \"kubernetes.io/projected/39b0c961-1859-4198-922c-5ff1780200a9-kube-api-access-ntx2w\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.030109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.029960 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.030109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.029979 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0c961-1859-4198-922c-5ff1780200a9-openshift-service-ca\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.030109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.030080 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-metrics-certs\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.030283 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:02:52.030113 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-serving-cert: secret "maas-api-e2e-ait-90774e61-serving-cert" not found Nov 03 19:02:52.030283 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:02:52.030191 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls podName:39b0c961-1859-4198-922c-5ff1780200a9 nodeName:}" failed. No retries permitted until 2026-11-03 19:02:52.530167853 +0000 UTC m=+1447.411583933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls") pod "maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" (UID: "39b0c961-1859-4198-922c-5ff1780200a9") : secret "maas-api-e2e-ait-90774e61-serving-cert" not found Nov 03 19:02:52.030817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.030787 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0c961-1859-4198-922c-5ff1780200a9-openshift-service-ca\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.032460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.032438 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-metrics-certs\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.091994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.091962 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntx2w\" (UniqueName: \"kubernetes.io/projected/39b0c961-1859-4198-922c-5ff1780200a9-kube-api-access-ntx2w\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.100446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.100405 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:52.317381 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.317339 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:52.523849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.523816 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:02:52.524023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.523877 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:02:52.532938 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.532908 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:52.537505 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.537480 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.539923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.539903 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls\") pod \"maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:52.793839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:52.793749 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:53.186793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:53.186768 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6"] Nov 03 19:02:53.188643 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:02:53.188598 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b0c961_1859_4198_922c_5ff1780200a9.slice/crio-4b81f54c309b0dd2a05862adc0349a5181587c620e6ceaa28702cf30c6cdc441 WatchSource:0}: Error finding container 4b81f54c309b0dd2a05862adc0349a5181587c620e6ceaa28702cf30c6cdc441: Status 404 returned error can't find the container with id 4b81f54c309b0dd2a05862adc0349a5181587c620e6ceaa28702cf30c6cdc441 Nov 03 19:02:53.523942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:53.523860 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:02:53.523942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:53.523919 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:02:53.594421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:53.594380 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" event={"ID":"39b0c961-1859-4198-922c-5ff1780200a9","Type":"ContainerStarted","Data":"4b81f54c309b0dd2a05862adc0349a5181587c620e6ceaa28702cf30c6cdc441"} Nov 03 19:02:54.523603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:54.523510 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:02:54.523603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:54.523585 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:02:54.603486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:54.603441 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" event={"ID":"39b0c961-1859-4198-922c-5ff1780200a9","Type":"ContainerStarted","Data":"7e33ab6fb7d6c11d973b1f931cb6e5b4ce94a963c86e280e46d5166563eee815"} Nov 03 19:02:54.603676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:54.603643 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:02:54.742317 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:54.742260 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" podStartSLOduration=2.79329017 podStartE2EDuration="3.742245754s" totalImagesPullingTime="948.955584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:54.739861794 +0000 UTC m=+1449.621277911" watchObservedRunningTime="2026-11-03 19:02:54.742245754 +0000 UTC m=+1449.623661856" Nov 03 19:02:55.523695 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:55.523641 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:02:55.524207 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:55.523730 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:02:55.938793 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:02:55.938758 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:02:56.525096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:56.525054 5988 patch_prober.go:29] interesting pod/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 19:02:56.525644 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:56.525135 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 19:02:57.620212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:57.620184 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:57.620999 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:57.620972 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:57.622489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:57.622454 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:02:57.659511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:57.659442 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:02:58.444610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:02:58.444567 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:00.617267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:00.617239 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:03:02.223465 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:02.223424 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:04.540386 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:04.540354 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:06.517864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.517834 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp"] Nov 03 19:03:06.522342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.522321 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.574125 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.574099 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6f198bfb-serving-cert\"" Nov 03 19:03:06.620998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.620968 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp"] Nov 03 19:03:06.654002 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.653971 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-maas-api-tls\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.654144 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.654123 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcp5w\" (UniqueName: \"kubernetes.io/projected/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-kube-api-access-qcp5w\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.654215 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.654182 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-openshift-service-ca\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.654452 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.654431 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-metrics-certs\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.755982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.755954 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-maas-api-tls\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.756126 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.756031 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcp5w\" (UniqueName: \"kubernetes.io/projected/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-kube-api-access-qcp5w\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.756126 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.756063 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-openshift-service-ca\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.756207 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.756175 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-metrics-certs\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.756739 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.756714 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-openshift-service-ca\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.758421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.758397 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-metrics-certs\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.758421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.758413 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-maas-api-tls\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.771342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.771272 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:06.813494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.813465 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcp5w\" (UniqueName: \"kubernetes.io/projected/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-kube-api-access-qcp5w\") pod \"maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.836162 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.836137 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:06.975551 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:06.975494 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:07.205142 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:07.205120 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp"] Nov 03 19:03:07.207077 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:03:07.207045 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c2fd8e_cf2c_493a_854b_d2e9b2c92a74.slice/crio-0a403271263a3fd4d8fcb1a04b03a5396bb5e273b448d84058f76228fd2e1a53 WatchSource:0}: Error finding container 0a403271263a3fd4d8fcb1a04b03a5396bb5e273b448d84058f76228fd2e1a53: Status 404 returned error can't find the container with id 0a403271263a3fd4d8fcb1a04b03a5396bb5e273b448d84058f76228fd2e1a53 Nov 03 19:03:07.691447 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:07.691409 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" event={"ID":"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74","Type":"ContainerStarted","Data":"0a403271263a3fd4d8fcb1a04b03a5396bb5e273b448d84058f76228fd2e1a53"} Nov 03 19:03:08.700993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:08.700940 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" event={"ID":"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74","Type":"ContainerStarted","Data":"e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21"} Nov 03 19:03:08.701534 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:08.701019 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:08.812826 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:08.812776 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" podStartSLOduration=1.904662594 podStartE2EDuration="2.812760808s" totalImagesPullingTime="908.098214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:08.811915883 +0000 UTC m=+1463.693332029" watchObservedRunningTime="2026-11-03 19:03:08.812760808 +0000 UTC m=+1463.694176910" Nov 03 19:03:11.213650 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.213618 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h"] Nov 03 19:03:11.218233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.218211 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.269933 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.269910 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-30b92815-serving-cert\"" Nov 03 19:03:11.317205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.317175 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dqz\" (UniqueName: \"kubernetes.io/projected/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-kube-api-access-q8dqz\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.317368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.317319 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-metrics-certs\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.317420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.317392 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-openshift-service-ca\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.317469 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.317418 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.319656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.319635 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h"] Nov 03 19:03:11.418378 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.418347 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dqz\" (UniqueName: \"kubernetes.io/projected/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-kube-api-access-q8dqz\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.418531 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.418466 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-metrics-certs\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.418573 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.418545 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-openshift-service-ca\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.418608 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.418580 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.418792 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:03:11.418774 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-serving-cert: secret "maas-api-e2e-derive-30b92815-serving-cert" not found Nov 03 19:03:11.418853 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:03:11.418828 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls podName:5e9f9e30-12bc-403b-b595-7a2f8ea2abcb nodeName:}" failed. No retries permitted until 2026-11-03 19:03:11.918813426 +0000 UTC m=+1466.800229510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls") pod "maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" (UID: "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb") : secret "maas-api-e2e-derive-30b92815-serving-cert" not found Nov 03 19:03:11.419228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.419203 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-openshift-service-ca\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.420883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.420864 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-metrics-certs\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.473995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.473904 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:11.476799 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.476766 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dqz\" (UniqueName: \"kubernetes.io/projected/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-kube-api-access-q8dqz\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.693377 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.693343 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:11.904175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.904127 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:11.925933 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.925901 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:11.928289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:11.928270 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls\") pod \"maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:12.132981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:12.132944 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:12.504933 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:12.504898 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h"] Nov 03 19:03:12.506655 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:03:12.506625 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9f9e30_12bc_403b_b595_7a2f8ea2abcb.slice/crio-929556b8401f33ccc2f517e7d67f00655595e22ccda21f529c485e06209bbbc4 WatchSource:0}: Error finding container 929556b8401f33ccc2f517e7d67f00655595e22ccda21f529c485e06209bbbc4: Status 404 returned error can't find the container with id 929556b8401f33ccc2f517e7d67f00655595e22ccda21f529c485e06209bbbc4 Nov 03 19:03:12.726515 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:12.726483 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" event={"ID":"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb","Type":"ContainerStarted","Data":"929556b8401f33ccc2f517e7d67f00655595e22ccda21f529c485e06209bbbc4"} Nov 03 19:03:13.735771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:13.735725 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" event={"ID":"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb","Type":"ContainerStarted","Data":"af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c"} Nov 03 19:03:13.736268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:13.735822 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:13.849566 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:13.849510 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" podStartSLOduration=2.071875862 podStartE2EDuration="2.849490939s" totalImagesPullingTime="777.615077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:13.84663359 +0000 UTC m=+1468.728049691" watchObservedRunningTime="2026-11-03 19:03:13.849490939 +0000 UTC m=+1468.730907042" Nov 03 19:03:14.712505 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:14.712477 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:03:14.929590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:14.929558 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:16.235244 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:16.235198 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:19.257422 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:19.255762 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.257422 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:19.255917 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 19:03:19.752708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:19.750742 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.769219 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:19.769187 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:03:19.816828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:19.816758 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:19.939702 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:19.939112 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.249708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.224440 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.297716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.257770 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.297716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.258137 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 19:03:20.297716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.258451 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:03:20.297716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.258508 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:03:20.330003 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.317822 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.330003 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.318204 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 03 19:03:20.358708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.341563 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.416705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.359004 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.762595 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.762550 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.765893 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.765865 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:03:20.803740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.803716 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:03:20.819807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.819766 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:03:20.821137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:20.821116 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:03:21.112195 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:21.112160 5988 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Nov 03 19:03:21.120516 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:21.119701 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:03:21.198281 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:21.185129 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:03:21.202800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:21.200790 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:03:22.035579 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:22.035524 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:23.314266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:23.314225 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:25.937725 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:03:25.937652 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:03:28.006306 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:28.006269 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:32.603958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:32.603906 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:37.203631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:37.203590 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:41.809711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:41.809642 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:46.517123 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.517084 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:46.684762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.684726 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb"] Nov 03 19:03:46.690859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.690836 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.740347 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.740321 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h6x4t\"" Nov 03 19:03:46.790760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.790672 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb"] Nov 03 19:03:46.791471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791440 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.791568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791476 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.791568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791561 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.791710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791673 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.791766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791729 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.791855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791839 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.791908 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791881 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.791982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.791963 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.792037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.792030 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86km\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-kube-api-access-v86km\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.893590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893560 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.893783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893608 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.893783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893652 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.893783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893744 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86km\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-kube-api-access-v86km\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.893959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893783 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.893959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893806 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.893959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893857 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.894107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.893969 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.894107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.894014 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.894374 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.894325 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.894374 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.894366 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.894545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.894390 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.894545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.894491 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.894634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.894561 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.896085 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.896063 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.896603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.896581 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.952619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.952583 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:46.952843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:46.952823 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86km\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-kube-api-access-v86km\") pod \"maas-default-gateway-openshift-default-7854949c9-2fxrb\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:47.004891 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:47.004858 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:47.457491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:47.457466 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb"] Nov 03 19:03:47.458652 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:03:47.458625 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efba1c7_6444_4ccc_94f3_4395a528bcaf.slice/crio-b15e73af9b0e0f620ba50dfb4916ea03e4e31be2792e7d8523bca661c3f8b2bb WatchSource:0}: Error finding container b15e73af9b0e0f620ba50dfb4916ea03e4e31be2792e7d8523bca661c3f8b2bb: Status 404 returned error can't find the container with id b15e73af9b0e0f620ba50dfb4916ea03e4e31be2792e7d8523bca661c3f8b2bb Nov 03 19:03:47.461841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:47.461806 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:03:47.461951 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:47.461886 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:03:47.462021 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:47.461959 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:03:48.100664 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:48.100624 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" event={"ID":"4efba1c7-6444-4ccc-94f3-4395a528bcaf","Type":"ContainerStarted","Data":"bb52d4d714127f0c9503747328c0a6b5e4562bb7efe2e4cc8cc0656cb395a6a2"} Nov 03 19:03:48.101086 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:48.100712 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" event={"ID":"4efba1c7-6444-4ccc-94f3-4395a528bcaf","Type":"ContainerStarted","Data":"b15e73af9b0e0f620ba50dfb4916ea03e4e31be2792e7d8523bca661c3f8b2bb"} Nov 03 19:03:48.216510 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:48.216449 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podStartSLOduration=2.216436909 podStartE2EDuration="2.216436909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:48.21445775 +0000 UTC m=+1503.095873864" watchObservedRunningTime="2026-11-03 19:03:48.216436909 +0000 UTC m=+1503.097853011" Nov 03 19:03:49.005174 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:49.005130 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:49.006837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:49.006812 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:03:49.006971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:49.006891 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:03:49.191137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:49.191098 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:50.005315 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:50.005277 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:03:50.005491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:50.005352 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:03:51.006052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:51.006019 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:03:51.006464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:51.006079 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:03:51.680174 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:51.680129 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:03:52.005471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:52.005386 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:03:52.005471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:52.005456 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:03:53.005425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:53.005389 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:03:53.005859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:53.005467 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:03:54.005631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:54.005597 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:03:54.006041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:54.005647 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:03:55.005459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:55.005422 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 19:03:55.005634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:55.005478 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 19:03:55.945875 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:03:55.945834 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:03:56.514631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:56.514599 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:03:56.706793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:56.706755 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:56.707307 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:56.707282 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:56.708138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:56.708119 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:03:57.652505 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:03:57.652463 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:01.807009 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.806969 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2"] Nov 03 19:04:01.812312 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.812290 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835202 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835177 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835210 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa252a91-d63e-439a-a035-605aff921960-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835243 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835323 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa252a91-d63e-439a-a035-605aff921960-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835363 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835391 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835522 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835641 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835616 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqwq6\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-kube-api-access-nqwq6\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.835751 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.835738 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.906588 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.906550 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2"] Nov 03 19:04:01.937420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937392 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqwq6\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-kube-api-access-nqwq6\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937569 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937464 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937569 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937544 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937735 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937572 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa252a91-d63e-439a-a035-605aff921960-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937735 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937603 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937735 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937643 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa252a91-d63e-439a-a035-605aff921960-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937735 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937710 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937972 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937749 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.937972 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937866 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.938079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937965 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.938079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.937993 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.938554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.938469 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.938957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.938925 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa252a91-d63e-439a-a035-605aff921960-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.939251 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.939227 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.940572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.940543 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.940716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.940605 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa252a91-d63e-439a-a035-605aff921960-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.995919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.995891 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:01.996032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:01.996021 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqwq6\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-kube-api-access-nqwq6\") pod \"maas-default-gateway-openshift-default-7854949c9-x4nc2\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:02.128604 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:02.128571 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:02.554595 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:02.554570 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2"] Nov 03 19:04:02.556009 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:04:02.555977 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa252a91_d63e_439a_a035_605aff921960.slice/crio-3ccc58f424035093771e083c19f56d5d6035afbc5d8cb180f77bee37ce70e8bd WatchSource:0}: Error finding container 3ccc58f424035093771e083c19f56d5d6035afbc5d8cb180f77bee37ce70e8bd: Status 404 returned error can't find the container with id 3ccc58f424035093771e083c19f56d5d6035afbc5d8cb180f77bee37ce70e8bd Nov 03 19:04:02.558678 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:02.558639 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:04:02.558767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:02.558746 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:04:02.558822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:02.558804 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:04:03.047652 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:03.047618 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:03.224990 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:03.224952 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" event={"ID":"aa252a91-d63e-439a-a035-605aff921960","Type":"ContainerStarted","Data":"980cd2e00d5c9b072e11e59555991513bcd07834a0ae0a36d6f9f6b78aff91f0"} Nov 03 19:04:03.225157 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:03.225051 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" event={"ID":"aa252a91-d63e-439a-a035-605aff921960","Type":"ContainerStarted","Data":"3ccc58f424035093771e083c19f56d5d6035afbc5d8cb180f77bee37ce70e8bd"} Nov 03 19:04:03.341310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:03.341263 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podStartSLOduration=2.341249623 podStartE2EDuration="2.341249623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:03.339836132 +0000 UTC m=+1518.221252240" watchObservedRunningTime="2026-11-03 19:04:03.341249623 +0000 UTC m=+1518.222665724" Nov 03 19:04:04.128944 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:04.128912 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:04.130832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:04.130804 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:04:04.130965 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:04.130850 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:04:05.130592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:05.130562 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:04:05.131006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:05.130616 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:04:06.128954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:06.128923 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:04:06.129162 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:06.128973 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:04:06.601315 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:06.601274 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:06.814610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:06.814555 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:07.129258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:07.129224 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:04:07.129574 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:07.129279 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:04:08.129987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:08.129949 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:04:08.130597 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:08.130568 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:04:08.162355 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:08.162320 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:09.129716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:09.129666 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:04:09.129922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:09.129748 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:04:10.129844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:10.129805 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 19:04:10.130254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:10.129887 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 19:04:11.134197 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:11.134165 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:11.134653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:11.134634 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:11.135343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:11.135325 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:04:11.458069 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:11.457957 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:11.668993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:11.668953 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:12.922537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:12.922501 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:13.665711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:13.665637 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:18.695701 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:18.695622 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:21.778543 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:21.778509 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4"] Nov 03 19:04:22.264352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:22.264294 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:22.479387 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:22.479350 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:23.621520 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:23.621473 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:25.943188 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:04:25.943151 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:04:26.779136 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:26.779093 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" podUID="dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" containerName="payload-processing" containerID="cri-o://c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7" gracePeriod=25 Nov 03 19:04:27.058371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.058343 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:04:27.125394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.125363 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-plugins-config-volume\") pod \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " Nov 03 19:04:27.125590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.125573 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-kube-api-access-6fqbp\" (UniqueName: \"kubernetes.io/projected/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-kube-api-access-6fqbp\") pod \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\" (UID: \"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15\") " Nov 03 19:04:27.125740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.125674 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-plugins-config-volume" pod "dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" (UID: "dfcb07b3-c2a2-4074-b1e5-ed4c73121c15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:27.126172 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.126143 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:27.127632 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.127605 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-kube-api-access-6fqbp" pod "dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" (UID: "dfcb07b3-c2a2-4074-b1e5-ed4c73121c15"). InnerVolumeSpecName "kube-api-access-6fqbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:27.227590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.227559 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fqbp\" (UniqueName: \"kubernetes.io/projected/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15-kube-api-access-6fqbp\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:27.421597 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.421558 5988 generic.go:356] "Generic (PLEG): container finished" podID="dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" containerID="c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7" exitCode=0 Nov 03 19:04:27.421788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.421597 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" Nov 03 19:04:27.421788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.421636 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" event={"ID":"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15","Type":"ContainerDied","Data":"c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7"} Nov 03 19:04:27.421788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.421707 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4" event={"ID":"dfcb07b3-c2a2-4074-b1e5-ed4c73121c15","Type":"ContainerDied","Data":"0639dfa85ab32fdc201122864203e92e297e9cc48f3d9ceca7f036e96162a006"} Nov 03 19:04:27.421788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.421737 5988 scope.go:122] "RemoveContainer" containerID="c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7" Nov 03 19:04:27.432205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.432183 5988 scope.go:122] "RemoveContainer" containerID="c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7" Nov 03 19:04:27.432507 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:04:27.432487 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7\": container with ID starting with c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7 not found: ID does not exist" containerID="c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7" Nov 03 19:04:27.432594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.432519 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7"} err="failed to get container status \"c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7\": rpc error: code = NotFound desc = could not find container \"c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7\": container with ID starting with c79a7490bb2b5aeb9e62406954337ebfb2956dccaa1f740f893b2b6229d22ef7 not found: ID does not exist" Nov 03 19:04:27.576935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.576898 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4"] Nov 03 19:04:27.583310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.583274 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6f198bfb-7dccb8fbcf-r24t4"] Nov 03 19:04:27.905542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:27.905506 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:28.118111 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:28.118072 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:28.333197 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:28.333145 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:29.281237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:29.281200 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" path="/var/lib/kubelet/pods/dfcb07b3-c2a2-4074-b1e5-ed4c73121c15/volumes" Nov 03 19:04:32.940651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:32.940601 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:37.522946 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:37.522902 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:39.993603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:39.993570 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:41.391210 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.391162 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6"] Nov 03 19:04:41.391704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.391509 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" podUID="39b0c961-1859-4198-922c-5ff1780200a9" containerName="maas-api" containerID="cri-o://7e33ab6fb7d6c11d973b1f931cb6e5b4ce94a963c86e280e46d5166563eee815" gracePeriod=30 Nov 03 19:04:41.527746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.527716 5988 generic.go:356] "Generic (PLEG): container finished" podID="39b0c961-1859-4198-922c-5ff1780200a9" containerID="7e33ab6fb7d6c11d973b1f931cb6e5b4ce94a963c86e280e46d5166563eee815" exitCode=0 Nov 03 19:04:41.527879 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.527790 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" event={"ID":"39b0c961-1859-4198-922c-5ff1780200a9","Type":"ContainerDied","Data":"7e33ab6fb7d6c11d973b1f931cb6e5b4ce94a963c86e280e46d5166563eee815"} Nov 03 19:04:41.556601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.556548 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:41.763959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.763929 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:04:41.840343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.840305 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b0c961-1859-4198-922c-5ff1780200a9-kube-api-access-ntx2w\" (UniqueName: \"kubernetes.io/projected/39b0c961-1859-4198-922c-5ff1780200a9-kube-api-access-ntx2w\") pod \"39b0c961-1859-4198-922c-5ff1780200a9\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " Nov 03 19:04:41.840622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.840600 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-metrics-certs\") pod \"39b0c961-1859-4198-922c-5ff1780200a9\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " Nov 03 19:04:41.840881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.840862 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b0c961-1859-4198-922c-5ff1780200a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0c961-1859-4198-922c-5ff1780200a9-openshift-service-ca\") pod \"39b0c961-1859-4198-922c-5ff1780200a9\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " Nov 03 19:04:41.841135 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.841115 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls\") pod \"39b0c961-1859-4198-922c-5ff1780200a9\" (UID: \"39b0c961-1859-4198-922c-5ff1780200a9\") " Nov 03 19:04:41.843194 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.843165 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b0c961-1859-4198-922c-5ff1780200a9-openshift-service-ca" pod "39b0c961-1859-4198-922c-5ff1780200a9" (UID: "39b0c961-1859-4198-922c-5ff1780200a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:41.860732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.858871 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b0c961-1859-4198-922c-5ff1780200a9-kube-api-access-ntx2w" pod "39b0c961-1859-4198-922c-5ff1780200a9" (UID: "39b0c961-1859-4198-922c-5ff1780200a9"). InnerVolumeSpecName "kube-api-access-ntx2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:41.860732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.859000 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-metrics-certs" pod "39b0c961-1859-4198-922c-5ff1780200a9" (UID: "39b0c961-1859-4198-922c-5ff1780200a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:41.861251 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.861208 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls" pod "39b0c961-1859-4198-922c-5ff1780200a9" (UID: "39b0c961-1859-4198-922c-5ff1780200a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:41.943175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.943140 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntx2w\" (UniqueName: \"kubernetes.io/projected/39b0c961-1859-4198-922c-5ff1780200a9-kube-api-access-ntx2w\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:41.943175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.943170 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:41.943175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.943180 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0c961-1859-4198-922c-5ff1780200a9-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:41.943485 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:41.943188 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39b0c961-1859-4198-922c-5ff1780200a9-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:42.536989 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:42.536953 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" event={"ID":"39b0c961-1859-4198-922c-5ff1780200a9","Type":"ContainerDied","Data":"4b81f54c309b0dd2a05862adc0349a5181587c620e6ceaa28702cf30c6cdc441"} Nov 03 19:04:42.537459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:42.537028 5988 scope.go:122] "RemoveContainer" containerID="7e33ab6fb7d6c11d973b1f931cb6e5b4ce94a963c86e280e46d5166563eee815" Nov 03 19:04:42.537459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:42.536964 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6" Nov 03 19:04:42.695253 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:42.695221 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6"] Nov 03 19:04:42.699634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:42.699602 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-90774e61-6bd7dcbc96-pjdh6"] Nov 03 19:04:43.062522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:43.062490 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:43.274226 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:43.274195 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b0c961-1859-4198-922c-5ff1780200a9" path="/var/lib/kubelet/pods/39b0c961-1859-4198-922c-5ff1780200a9/volumes" Nov 03 19:04:43.322064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:43.321888 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:43.568855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:43.568822 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:43.850343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:43.850313 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:44.586112 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:44.586074 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:44.803081 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:44.803047 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:45.011176 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:45.011092 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:45.449706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:45.448948 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:45.668733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:45.668661 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:45.895377 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:45.895335 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:46.606631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:46.606584 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:46.820778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:46.820661 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:47.832398 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:47.832339 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:48.939791 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:48.939756 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:49.554591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:49.554556 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:49.755920 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:49.755885 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:50.022987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:50.022957 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:52.161094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.161063 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:52.389752 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.389712 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:52.604865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.604830 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:52.858954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.858867 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq"] Nov 03 19:04:52.859619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.859596 5988 cpu_manager.go:414] "containerMap: removing container" podUID="dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" containerName="payload-processing" Nov 03 19:04:52.859619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.859619 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" containerName="payload-processing" Nov 03 19:04:52.859841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.859649 5988 cpu_manager.go:414] "containerMap: removing container" podUID="39b0c961-1859-4198-922c-5ff1780200a9" containerName="maas-api" Nov 03 19:04:52.859841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.859658 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b0c961-1859-4198-922c-5ff1780200a9" containerName="maas-api" Nov 03 19:04:52.859958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.859842 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcb07b3-c2a2-4074-b1e5-ed4c73121c15" containerName="payload-processing" Nov 03 19:04:52.859958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.859861 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b0c961-1859-4198-922c-5ff1780200a9" containerName="maas-api" Nov 03 19:04:52.864789 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.864768 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:52.914072 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.914043 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-82cd41ab\"" Nov 03 19:04:52.918667 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.918639 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq"] Nov 03 19:04:52.954794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.954755 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894746b-3cc6-46d2-ad6d-c9aa304e132e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:52.955087 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.954863 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqk67\" (UniqueName: \"kubernetes.io/projected/c894746b-3cc6-46d2-ad6d-c9aa304e132e-kube-api-access-gqk67\") pod \"payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:52.960563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:52.960536 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-82cd41ab-dockercfg-crplt\"" Nov 03 19:04:53.056351 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.056304 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894746b-3cc6-46d2-ad6d-c9aa304e132e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:53.056575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.056385 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqk67\" (UniqueName: \"kubernetes.io/projected/c894746b-3cc6-46d2-ad6d-c9aa304e132e-kube-api-access-gqk67\") pod \"payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:53.057198 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.057164 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894746b-3cc6-46d2-ad6d-c9aa304e132e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:53.114246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.114184 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqk67\" (UniqueName: \"kubernetes.io/projected/c894746b-3cc6-46d2-ad6d-c9aa304e132e-kube-api-access-gqk67\") pod \"payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:53.179920 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.179867 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:53.557785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.557707 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp"] Nov 03 19:04:53.558004 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.557980 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" podUID="72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" containerName="maas-api" containerID="cri-o://e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21" gracePeriod=30 Nov 03 19:04:53.563614 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.563439 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq"] Nov 03 19:04:53.565774 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:04:53.565749 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc894746b_3cc6_46d2_ad6d_c9aa304e132e.slice/crio-66161b1c86b499299b0fe4f46332e0b51cb14fbddb51b2a11a2f39fae2609e82 WatchSource:0}: Error finding container 66161b1c86b499299b0fe4f46332e0b51cb14fbddb51b2a11a2f39fae2609e82: Status 404 returned error can't find the container with id 66161b1c86b499299b0fe4f46332e0b51cb14fbddb51b2a11a2f39fae2609e82 Nov 03 19:04:53.633335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.633295 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" event={"ID":"c894746b-3cc6-46d2-ad6d-c9aa304e132e","Type":"ContainerStarted","Data":"66161b1c86b499299b0fe4f46332e0b51cb14fbddb51b2a11a2f39fae2609e82"} Nov 03 19:04:53.734106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.734081 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:53.874924 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.874875 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:04:53.976201 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.976166 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-openshift-service-ca\") pod \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " Nov 03 19:04:53.976412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.976276 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-maas-api-tls\") pod \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " Nov 03 19:04:53.976412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.976327 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-kube-api-access-qcp5w\" (UniqueName: \"kubernetes.io/projected/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-kube-api-access-qcp5w\") pod \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " Nov 03 19:04:53.976553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.976477 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-metrics-certs\") pod \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\" (UID: \"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74\") " Nov 03 19:04:53.976605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.976576 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-openshift-service-ca" pod "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" (UID: "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:53.977325 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.977263 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:53.979407 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.979373 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-maas-api-tls" pod "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" (UID: "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:53.979717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.979644 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-metrics-certs" pod "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" (UID: "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:53.979850 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:53.979827 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-kube-api-access-qcp5w" pod "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" (UID: "72c2fd8e-cf2c-493a-854b-d2e9b2c92a74"). InnerVolumeSpecName "kube-api-access-qcp5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:54.085373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.079335 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcp5w\" (UniqueName: \"kubernetes.io/projected/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-kube-api-access-qcp5w\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:54.085373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.079374 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:54.085373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.079389 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:54.681363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.681327 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" event={"ID":"c894746b-3cc6-46d2-ad6d-c9aa304e132e","Type":"ContainerStarted","Data":"df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c"} Nov 03 19:04:54.681922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.681776 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:04:54.683666 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.683630 5988 generic.go:356] "Generic (PLEG): container finished" podID="72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" containerID="e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21" exitCode=0 Nov 03 19:04:54.683832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.683725 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" event={"ID":"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74","Type":"ContainerDied","Data":"e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21"} Nov 03 19:04:54.683832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.683768 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" event={"ID":"72c2fd8e-cf2c-493a-854b-d2e9b2c92a74","Type":"ContainerDied","Data":"0a403271263a3fd4d8fcb1a04b03a5396bb5e273b448d84058f76228fd2e1a53"} Nov 03 19:04:54.683832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.683796 5988 scope.go:122] "RemoveContainer" containerID="e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21" Nov 03 19:04:54.684000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.683941 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp" Nov 03 19:04:54.699464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.699442 5988 scope.go:122] "RemoveContainer" containerID="e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21" Nov 03 19:04:54.699765 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:04:54.699737 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21\": container with ID starting with e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21 not found: ID does not exist" containerID="e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21" Nov 03 19:04:54.699849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.699774 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21"} err="failed to get container status \"e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21\": rpc error: code = NotFound desc = could not find container \"e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21\": container with ID starting with e6f73b0593734299b1f61bad78760b190323169869ca455332043e241f9e4a21 not found: ID does not exist" Nov 03 19:04:54.792956 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.792343 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" podStartSLOduration=2.792324365 podStartE2EDuration="2.792324365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:54.791180277 +0000 UTC m=+1569.672596380" watchObservedRunningTime="2026-11-03 19:04:54.792324365 +0000 UTC m=+1569.673740468" Nov 03 19:04:54.898164 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.898126 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp"] Nov 03 19:04:54.900922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.900880 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f198bfb-858cc7584f-bbkzp"] Nov 03 19:04:54.903726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.903705 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h"] Nov 03 19:04:54.903916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:54.903896 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" podUID="5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" containerName="maas-api" containerID="cri-o://af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c" gracePeriod=30 Nov 03 19:04:55.068575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.068544 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:55.239396 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.239149 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:04:55.275309 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.275267 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" path="/var/lib/kubelet/pods/72c2fd8e-cf2c-493a-854b-d2e9b2c92a74/volumes" Nov 03 19:04:55.306845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.306815 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-openshift-service-ca\") pod \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " Nov 03 19:04:55.307062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.307039 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-metrics-certs\") pod \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " Nov 03 19:04:55.307148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.307078 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls\") pod \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " Nov 03 19:04:55.307205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.307150 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-kube-api-access-q8dqz\" (UniqueName: \"kubernetes.io/projected/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-kube-api-access-q8dqz\") pod \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\" (UID: \"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb\") " Nov 03 19:04:55.307266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.307209 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-openshift-service-ca" pod "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" (UID: "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:04:55.307844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.307816 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:55.309209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.309187 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-metrics-certs" pod "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" (UID: "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:55.309642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.309611 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls" pod "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" (UID: "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:04:55.309642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.309614 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-kube-api-access-q8dqz" pod "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" (UID: "5e9f9e30-12bc-403b-b595-7a2f8ea2abcb"). InnerVolumeSpecName "kube-api-access-q8dqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:04:55.412080 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.411903 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:55.412290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.412270 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:55.412418 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.412403 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8dqz\" (UniqueName: \"kubernetes.io/projected/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb-kube-api-access-q8dqz\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:04:55.690949 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.690918 5988 generic.go:356] "Generic (PLEG): container finished" podID="5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" containerID="af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c" exitCode=0 Nov 03 19:04:55.691437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.690978 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" Nov 03 19:04:55.691437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.690994 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" event={"ID":"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb","Type":"ContainerDied","Data":"af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c"} Nov 03 19:04:55.691437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.691070 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h" event={"ID":"5e9f9e30-12bc-403b-b595-7a2f8ea2abcb","Type":"ContainerDied","Data":"929556b8401f33ccc2f517e7d67f00655595e22ccda21f529c485e06209bbbc4"} Nov 03 19:04:55.691437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.691092 5988 scope.go:122] "RemoveContainer" containerID="af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c" Nov 03 19:04:55.701440 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.701419 5988 scope.go:122] "RemoveContainer" containerID="af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c" Nov 03 19:04:55.701732 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:04:55.701708 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c\": container with ID starting with af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c not found: ID does not exist" containerID="af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c" Nov 03 19:04:55.701786 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.701742 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c"} err="failed to get container status \"af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c\": rpc error: code = NotFound desc = could not find container \"af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c\": container with ID starting with af8ec217b62a9ba42cd14f503d9aa141036f2b8ad0633821805f6f107b5a2d7c not found: ID does not exist" Nov 03 19:04:55.845537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.845512 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h"] Nov 03 19:04:55.849846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.849826 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-30b92815-df5d5f78b-dvh8h"] Nov 03 19:04:55.892883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:55.892858 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:55.967760 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:04:55.942008 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:04:56.246023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:56.245919 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:57.081597 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:57.081559 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:57.274631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:57.274601 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" path="/var/lib/kubelet/pods/5e9f9e30-12bc-403b-b595-7a2f8ea2abcb/volumes" Nov 03 19:04:57.288394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:57.288353 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:57.498015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:57.497938 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:57.712583 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:57.712543 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:58.096619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.094848 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:58.353758 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.341444 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:58.581581 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.581546 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:58.790523 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.790419 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:58.946280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.946240 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm"] Nov 03 19:04:58.946966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.946947 5988 cpu_manager.go:414] "containerMap: removing container" podUID="5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" containerName="maas-api" Nov 03 19:04:58.947105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.946968 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" containerName="maas-api" Nov 03 19:04:58.947105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.947011 5988 cpu_manager.go:414] "containerMap: removing container" podUID="72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" containerName="maas-api" Nov 03 19:04:58.947105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.947020 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" containerName="maas-api" Nov 03 19:04:58.947316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.947222 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9f9e30-12bc-403b-b595-7a2f8ea2abcb" containerName="maas-api" Nov 03 19:04:58.947316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.947247 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c2fd8e-cf2c-493a-854b-d2e9b2c92a74" containerName="maas-api" Nov 03 19:04:58.953491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:58.953467 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.013750 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.012163 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm"] Nov 03 19:04:59.013750 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.013476 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:59.050670 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.050587 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9e277a\"" Nov 03 19:04:59.050670 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.050644 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9e277a-dockercfg-vpd29\"" Nov 03 19:04:59.097082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.097039 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.097567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.097125 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qckq\" (UniqueName: \"kubernetes.io/projected/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-kube-api-access-7qckq\") pod \"payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.199368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.199335 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.199653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.199629 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qckq\" (UniqueName: \"kubernetes.io/projected/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-kube-api-access-7qckq\") pod \"payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.200458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.200302 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.265203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.265166 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qckq\" (UniqueName: \"kubernetes.io/projected/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-kube-api-access-7qckq\") pod \"payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.269672 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.269638 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:04:59.322285 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.322256 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq"] Nov 03 19:04:59.328376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.328353 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.413778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.404062 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq"] Nov 03 19:04:59.509997 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.509963 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beb55de-3f42-4add-98e2-123ecfe00075-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.510176 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.510033 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtlq\" (UniqueName: \"kubernetes.io/projected/3beb55de-3f42-4add-98e2-123ecfe00075-kube-api-access-shtlq\") pod \"payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.611745 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.611651 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beb55de-3f42-4add-98e2-123ecfe00075-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.611917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.611747 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtlq\" (UniqueName: \"kubernetes.io/projected/3beb55de-3f42-4add-98e2-123ecfe00075-kube-api-access-shtlq\") pod \"payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.612412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.612385 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beb55de-3f42-4add-98e2-123ecfe00075-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.659512 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.659483 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52"] Nov 03 19:04:59.664039 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.664015 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.670272 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.670245 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtlq\" (UniqueName: \"kubernetes.io/projected/3beb55de-3f42-4add-98e2-123ecfe00075-kube-api-access-shtlq\") pod \"payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.698170 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.698140 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm"] Nov 03 19:04:59.743702 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.743605 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6765e083-openshift-default-dockercfg-jwgwn\"" Nov 03 19:04:59.747840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.747816 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52"] Nov 03 19:04:59.757642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.757619 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" event={"ID":"81c94319-5dd5-4e55-ae3f-f5f96e3f2778","Type":"ContainerStarted","Data":"bc354f4b26d7f4082e23e4d6954b9f213f09ced77440d638aac70f84bee630c7"} Nov 03 19:04:59.823102 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823067 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-envoy\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.823272 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823117 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istiod-ca-cert\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.823272 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823162 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-certs\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.823272 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823233 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfgh\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-kube-api-access-cgfgh\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.823450 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823354 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-data\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.823450 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823438 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-token\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.823552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823470 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-podinfo\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.823598 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.823546 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-socket\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.826629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.826594 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-credential-socket\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.827316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.827280 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:04:59.932091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932007 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-data\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932088 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-token\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932322 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932113 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-podinfo\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932322 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932167 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-socket\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932322 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932237 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-credential-socket\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932333 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-envoy\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932356 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istiod-ca-cert\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932389 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-certs\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932457 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfgh\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-kube-api-access-cgfgh\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932455 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-data\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932790 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932504 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-socket\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932892 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932870 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-credential-socket\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.932954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.932897 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-certs\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.933340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.933321 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istiod-ca-cert\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.934662 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.934638 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-envoy\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.934876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.934852 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-podinfo\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.948489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.948466 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:04:59.989476 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.989450 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-token\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.989724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.989676 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfgh\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-kube-api-access-cgfgh\") pod \"e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:04:59.992585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:04:59.992556 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:05:00.170599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.170558 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:00.328974 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.328944 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq"] Nov 03 19:05:00.364372 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:05:00.364340 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beb55de_3f42_4add_98e2_123ecfe00075.slice/crio-0300a7f164c1e73f81b15ccfc8be642e0bb6fd1724e89cd8ecc02680166a0700 WatchSource:0}: Error finding container 0300a7f164c1e73f81b15ccfc8be642e0bb6fd1724e89cd8ecc02680166a0700: Status 404 returned error can't find the container with id 0300a7f164c1e73f81b15ccfc8be642e0bb6fd1724e89cd8ecc02680166a0700 Nov 03 19:05:00.391441 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:05:00.391400 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ca1f38_4563_4ce9_b0fa_77c12e34b8bf.slice/crio-6f7983d7fe3ecb97ee5e794c75082c2842cc38cf993f632ee55899d1d51c4ed4 WatchSource:0}: Error finding container 6f7983d7fe3ecb97ee5e794c75082c2842cc38cf993f632ee55899d1d51c4ed4: Status 404 returned error can't find the container with id 6f7983d7fe3ecb97ee5e794c75082c2842cc38cf993f632ee55899d1d51c4ed4 Nov 03 19:05:00.391550 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.391494 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52"] Nov 03 19:05:00.394208 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.394177 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:00.394293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.394238 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:00.394293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.394264 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:00.765905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.765805 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" event={"ID":"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf","Type":"ContainerStarted","Data":"d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4"} Nov 03 19:05:00.765905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.765873 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" event={"ID":"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf","Type":"ContainerStarted","Data":"6f7983d7fe3ecb97ee5e794c75082c2842cc38cf993f632ee55899d1d51c4ed4"} Nov 03 19:05:00.767725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.767676 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" event={"ID":"81c94319-5dd5-4e55-ae3f-f5f96e3f2778","Type":"ContainerStarted","Data":"83ca87c52bd63bd1014d992ad3c61b13092b11f9de25da85fe3e3fed0af38742"} Nov 03 19:05:00.767901 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.767884 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:05:00.769736 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.769710 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" event={"ID":"3beb55de-3f42-4add-98e2-123ecfe00075","Type":"ContainerStarted","Data":"56b6af758cde5cce7c95ded6329998e3707aa6f67196cba6f91a9dfa7633fc4d"} Nov 03 19:05:00.769846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.769752 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" event={"ID":"3beb55de-3f42-4add-98e2-123ecfe00075","Type":"ContainerStarted","Data":"0300a7f164c1e73f81b15ccfc8be642e0bb6fd1724e89cd8ecc02680166a0700"} Nov 03 19:05:00.769846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.769799 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:05:00.880701 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.880644 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podStartSLOduration=1.880630249 podStartE2EDuration="1.880630249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:00.879924015 +0000 UTC m=+1575.761340116" watchObservedRunningTime="2026-11-03 19:05:00.880630249 +0000 UTC m=+1575.762046351" Nov 03 19:05:00.990713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.990632 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" podStartSLOduration=2.990615826 podStartE2EDuration="2.990615826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:00.988440808 +0000 UTC m=+1575.869856901" watchObservedRunningTime="2026-11-03 19:05:00.990615826 +0000 UTC m=+1575.872031933" Nov 03 19:05:00.992868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:00.992833 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:05:01.096518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:01.096451 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" podStartSLOduration=2.096431461 podStartE2EDuration="2.096431461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:01.096188417 +0000 UTC m=+1575.977604525" watchObservedRunningTime="2026-11-03 19:05:01.096431461 +0000 UTC m=+1575.977847563" Nov 03 19:05:01.999466 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:01.999431 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:05:02.798607 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:02.798576 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:05:02.800046 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:02.800005 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:02.800396 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:02.800374 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:05:03.009426 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:03.009392 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:03.227444 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:03.227409 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:05.695535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:05.695504 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:05:06.270521 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:06.270491 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:06.478642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:06.478605 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:08.122869 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:08.122827 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:09.858784 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.858756 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b"] Nov 03 19:05:09.863953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.863932 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.894293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.894269 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.894439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.894315 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.894510 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.894458 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2kg\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-kube-api-access-hc2kg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.894588 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.894570 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.894646 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.894606 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.894746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.894726 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.894852 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.894838 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.912837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.912807 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 19:05:09.913535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.913505 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 19:05:09.913602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.913592 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 19:05:09.962479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.962457 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b"] Nov 03 19:05:09.996039 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.996004 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.996183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.996110 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.996183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.996141 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.996314 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.996215 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2kg\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-kube-api-access-hc2kg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.996314 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.996285 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.996418 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.996315 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.996418 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.996373 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.997079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.997029 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.999168 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.999140 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.999267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.999245 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.999602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.999577 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:09.999948 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:09.999925 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:10.018328 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:10.018309 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4qf6p\"" Nov 03 19:05:10.053892 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:10.053867 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:10.054176 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:10.054156 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2kg\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-kube-api-access-hc2kg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xl65b\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:10.176749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:10.176636 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:10.651970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:10.651942 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b"] Nov 03 19:05:10.654058 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:05:10.654027 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad41e61_e71e_4735_9c7d_a07fbf7df6ad.slice/crio-e719dd464e4642fd3a30530360d59f98ee2544964a674524f76b915c28983ad0 WatchSource:0}: Error finding container e719dd464e4642fd3a30530360d59f98ee2544964a674524f76b915c28983ad0: Status 404 returned error can't find the container with id e719dd464e4642fd3a30530360d59f98ee2544964a674524f76b915c28983ad0 Nov 03 19:05:10.656134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:10.656114 5988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:05:10.872298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:10.872256 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" event={"ID":"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad","Type":"ContainerStarted","Data":"e719dd464e4642fd3a30530360d59f98ee2544964a674524f76b915c28983ad0"} Nov 03 19:05:11.779252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:11.779222 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:05:11.779467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:11.779298 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:05:12.821284 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:12.821224 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:14.353565 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:14.353525 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:05:14.635575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:14.635471 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:14.635575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:14.635542 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:14.903544 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:14.903449 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" event={"ID":"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad","Type":"ContainerStarted","Data":"7e00eb8b6d8861ea8211d2ec17061ba3d876149ccf0ca2abb47e89d596c2d84c"} Nov 03 19:05:14.903544 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:14.903518 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:15.085753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:15.073164 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" podStartSLOduration=2.095239127 podStartE2EDuration="6.07314307s" totalImagesPullingTime="3.977903943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:15.072020505 +0000 UTC m=+1589.953436608" watchObservedRunningTime="2026-11-03 19:05:15.07314307 +0000 UTC m=+1589.954559173" Nov 03 19:05:15.939093 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:15.939047 5988 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xl65b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:05:15.943492 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:15.942349 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" podUID="4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:05:17.487708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:17.487430 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:18.916668 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:18.916637 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:05:22.096116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:22.095394 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:25.936754 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:05:25.936713 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:05:26.703511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:26.703476 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:31.192875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:31.192827 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:31.404754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:31.403365 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:31.669492 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:31.669458 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:36.042178 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:36.042142 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:40.680364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:40.680323 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:40.898569 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:40.898534 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:41.211412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:41.211373 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:45.680313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:45.680276 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:47.653692 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.653659 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls"] Nov 03 19:05:47.662888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.662866 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.734881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.734842 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.734990 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.735106 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591c9108-119c-40d3-b301-cd6761ecb0dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735282 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.735165 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735282 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.735260 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.735397 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.735437 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735529 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.735476 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.735529 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.735503 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df488\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-kube-api-access-df488\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.784286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.784256 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls"] Nov 03 19:05:47.837206 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837174 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837264 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591c9108-119c-40d3-b301-cd6761ecb0dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837296 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837365 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837503 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837486 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837560 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837535 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837570 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837597 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df488\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-kube-api-access-df488\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837657 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837798 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.837886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.837862 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.838084 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.838062 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.838204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.838085 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591c9108-119c-40d3-b301-cd6761ecb0dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.838204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.838138 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.839594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.839574 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.839854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.839833 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.895436 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.895404 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.895637 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.895615 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df488\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-kube-api-access-df488\") pod \"maas-default-gateway-openshift-default-7854949c9-w6fls\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:47.978112 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:47.978047 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:48.399363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:48.399339 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls"] Nov 03 19:05:48.403361 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:48.403327 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:48.403448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:48.403392 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:48.403448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:48.403421 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:05:49.136509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:49.136472 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" event={"ID":"591c9108-119c-40d3-b301-cd6761ecb0dd","Type":"ContainerStarted","Data":"55e5d05cd7e1f5c32c1bedc7fba865c7fed57ef4a3d0d2e0449b33351e9a911a"} Nov 03 19:05:49.136915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:49.136536 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" event={"ID":"591c9108-119c-40d3-b301-cd6761ecb0dd","Type":"ContainerStarted","Data":"3d54a489c822092864ffb29f1f1fd76cd184e367d16a5ec06236cf6134558fd5"} Nov 03 19:05:49.269705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:49.266656 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podStartSLOduration=2.266636193 podStartE2EDuration="2.266636193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:49.264144679 +0000 UTC m=+1624.145560793" watchObservedRunningTime="2026-11-03 19:05:49.266636193 +0000 UTC m=+1624.148052296" Nov 03 19:05:49.941791 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:49.941756 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:49.978731 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:49.978702 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:49.980394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:49.980370 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:05:49.980497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:49.980417 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:05:50.189260 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:50.186149 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:50.404099 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:50.404068 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:50.615097 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:50.615060 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:50.979252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:50.979213 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:05:50.979521 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:50.979273 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:05:51.978664 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:51.978624 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:05:51.979085 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:51.978703 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:05:52.145672 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:52.145634 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:52.978658 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:52.978618 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:05:52.979153 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:52.978671 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:05:53.131497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:53.131431 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:53.366706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:53.363470 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:53.978995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:53.978956 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:05:53.979411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:53.979019 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:05:54.978411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:54.978381 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:05:54.978580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:54.978447 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:05:55.559369 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:55.559335 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:55.938287 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:05:55.938245 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:05:55.978394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:55.978366 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 19:05:55.978557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:55.978409 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 19:05:56.983254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:56.983218 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:56.983730 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:56.983548 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:56.984309 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:56.984290 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:05:58.176668 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:58.176634 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:05:58.381575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:05:58.381543 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:00.155375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:00.155294 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:00.558725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:00.558547 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:01.461216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:01.461187 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:01.682401 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:01.682361 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:02.856778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:02.856739 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:03.072250 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:03.072214 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:05.548648 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:05.548602 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:08.446876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:08.446839 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:06:10.304962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:10.304928 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:10.562764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:10.555119 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:11.093707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:11.089862 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:11.525246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:11.525148 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:15.811706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:15.804368 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:16.693145 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:16.693101 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:16.926205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:16.926166 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:17.148498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:17.148447 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:18.839846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:18.839807 5988 patch_prober.go:29] interesting pod/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:06:18.840261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:18.839863 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:06:19.430621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:19.430544 5988 generic.go:356] "Generic (PLEG): container finished" podID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerID="d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4" exitCode=0 Nov 03 19:06:19.430830 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:19.430616 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" event={"ID":"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf","Type":"ContainerDied","Data":"d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4"} Nov 03 19:06:19.431101 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:19.431086 5988 scope.go:122] "RemoveContainer" containerID="d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4" Nov 03 19:06:19.432800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:19.432774 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:06:19.432882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:19.432854 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:06:19.432929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:19.432885 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:06:19.996190 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:19.996153 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:20.201393 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:20.201362 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:20.445103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:20.445063 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" event={"ID":"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf","Type":"ContainerStarted","Data":"929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab"} Nov 03 19:06:20.993340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:20.993301 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:06:20.993579 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:20.993560 5988 patch_prober.go:29] interesting pod/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:06:20.993628 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:20.993608 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:06:21.780377 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:21.780328 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:21.987396 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:21.987364 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:21.993577 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:21.993549 5988 patch_prober.go:29] interesting pod/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:06:21.993765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:21.993619 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:06:22.995106 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:22.995068 5988 patch_prober.go:29] interesting pod/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:06:22.995753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:22.995721 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:06:23.993437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:23.993395 5988 patch_prober.go:29] interesting pod/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:06:23.993615 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:23.993467 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:06:24.993218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:24.993187 5988 patch_prober.go:29] interesting pod/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 03 19:06:24.993632 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:24.993241 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 03 19:06:25.939865 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:06:25.939823 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:06:26.028463 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:26.028426 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:06:26.029279 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:26.028883 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:06:26.031477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:26.031453 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:06:26.889111 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:26.889069 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:28.166390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:28.166355 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:28.848758 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:28.848723 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:31.231518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:31.231474 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:35.615216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:35.615181 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:36.240443 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:36.240404 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:41.561002 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:41.560957 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:43.465819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:43.465756 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:43.921935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:43.921892 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:44.547728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:44.547652 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:44.954843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:44.954809 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:47.021032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:47.020962 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:48.647507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:48.646927 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq"] Nov 03 19:06:48.998530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:48.998434 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:49.219718 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:49.219668 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:49.429606 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:49.429569 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:50.315458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:50.315424 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:50.534713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:50.534051 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:53.402785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:53.402737 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:53.617255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:53.617218 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:53.647874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:53.647820 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" podUID="c894746b-3cc6-46d2-ad6d-c9aa304e132e" containerName="payload-pre-processing" containerID="cri-o://df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c" gracePeriod=25 Nov 03 19:06:54.030334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.030308 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:06:54.184536 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.184495 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c894746b-3cc6-46d2-ad6d-c9aa304e132e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894746b-3cc6-46d2-ad6d-c9aa304e132e-plugins-config-volume\") pod \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " Nov 03 19:06:54.184764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.184597 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c894746b-3cc6-46d2-ad6d-c9aa304e132e-kube-api-access-gqk67\" (UniqueName: \"kubernetes.io/projected/c894746b-3cc6-46d2-ad6d-c9aa304e132e-kube-api-access-gqk67\") pod \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\" (UID: \"c894746b-3cc6-46d2-ad6d-c9aa304e132e\") " Nov 03 19:06:54.185027 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.184997 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c894746b-3cc6-46d2-ad6d-c9aa304e132e-plugins-config-volume" pod "c894746b-3cc6-46d2-ad6d-c9aa304e132e" (UID: "c894746b-3cc6-46d2-ad6d-c9aa304e132e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:06:54.185333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.185307 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c894746b-3cc6-46d2-ad6d-c9aa304e132e-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:54.187051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.187025 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c894746b-3cc6-46d2-ad6d-c9aa304e132e-kube-api-access-gqk67" pod "c894746b-3cc6-46d2-ad6d-c9aa304e132e" (UID: "c894746b-3cc6-46d2-ad6d-c9aa304e132e"). InnerVolumeSpecName "kube-api-access-gqk67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:06:54.292899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.292642 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqk67\" (UniqueName: \"kubernetes.io/projected/c894746b-3cc6-46d2-ad6d-c9aa304e132e-kube-api-access-gqk67\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:06:54.748307 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.748275 5988 generic.go:356] "Generic (PLEG): container finished" podID="c894746b-3cc6-46d2-ad6d-c9aa304e132e" containerID="df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c" exitCode=0 Nov 03 19:06:54.748816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.748317 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" Nov 03 19:06:54.748816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.748356 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" event={"ID":"c894746b-3cc6-46d2-ad6d-c9aa304e132e","Type":"ContainerDied","Data":"df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c"} Nov 03 19:06:54.748816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.748417 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq" event={"ID":"c894746b-3cc6-46d2-ad6d-c9aa304e132e","Type":"ContainerDied","Data":"66161b1c86b499299b0fe4f46332e0b51cb14fbddb51b2a11a2f39fae2609e82"} Nov 03 19:06:54.748816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.748453 5988 scope.go:122] "RemoveContainer" containerID="df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c" Nov 03 19:06:54.758800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.758782 5988 scope.go:122] "RemoveContainer" containerID="df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c" Nov 03 19:06:54.759067 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:06:54.759049 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c\": container with ID starting with df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c not found: ID does not exist" containerID="df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c" Nov 03 19:06:54.759116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.759076 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c"} err="failed to get container status \"df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c\": rpc error: code = NotFound desc = could not find container \"df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c\": container with ID starting with df3998baa2080ef38f68df784d765c7abbb44e73678142b2f3e9d1e3fc6cb35c not found: ID does not exist" Nov 03 19:06:54.907156 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.907117 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq"] Nov 03 19:06:54.952935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:54.952895 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-82cd41ab-799f447d67-ps6pq"] Nov 03 19:06:55.275122 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:55.275086 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c894746b-3cc6-46d2-ad6d-c9aa304e132e" path="/var/lib/kubelet/pods/c894746b-3cc6-46d2-ad6d-c9aa304e132e/volumes" Nov 03 19:06:55.937566 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:06:55.937529 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:06:59.414273 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:59.405156 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:59.643915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:59.643848 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:06:59.855284 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:06:59.855201 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:04.843617 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:04.843573 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:09.495126 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:09.495043 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:13.071972 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:13.071929 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:13.287350 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:13.287316 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:14.218812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:14.218779 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:16.400803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:16.400771 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:07:18.730333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.730298 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp"] Nov 03 19:07:18.730948 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.730933 5988 cpu_manager.go:414] "containerMap: removing container" podUID="c894746b-3cc6-46d2-ad6d-c9aa304e132e" containerName="payload-pre-processing" Nov 03 19:07:18.730999 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.730948 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c894746b-3cc6-46d2-ad6d-c9aa304e132e" containerName="payload-pre-processing" Nov 03 19:07:18.731090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.731081 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c894746b-3cc6-46d2-ad6d-c9aa304e132e" containerName="payload-pre-processing" Nov 03 19:07:18.734181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.734162 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.783158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.783129 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5ac14ff6-serving-cert\"" Nov 03 19:07:18.805101 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.805055 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2803a5-571f-4ead-8192-876175cb040c-openshift-service-ca\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.805290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.805213 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8r7g\" (UniqueName: \"kubernetes.io/projected/ec2803a5-571f-4ead-8192-876175cb040c-kube-api-access-d8r7g\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.805290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.805248 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-metrics-certs\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.805496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.805472 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.832483 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.832445 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp"] Nov 03 19:07:18.906605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.906574 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.906812 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:07:18.906739 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-serving-cert: secret "maas-api-e2e-mt-5ac14ff6-serving-cert" not found Nov 03 19:07:18.906812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.906777 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2803a5-571f-4ead-8192-876175cb040c-openshift-service-ca\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.906901 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:07:18.906816 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls podName:ec2803a5-571f-4ead-8192-876175cb040c nodeName:}" failed. No retries permitted until 2026-11-03 19:07:19.406795868 +0000 UTC m=+1714.288211955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls") pod "maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" (UID: "ec2803a5-571f-4ead-8192-876175cb040c") : secret "maas-api-e2e-mt-5ac14ff6-serving-cert" not found Nov 03 19:07:18.906951 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.906915 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8r7g\" (UniqueName: \"kubernetes.io/projected/ec2803a5-571f-4ead-8192-876175cb040c-kube-api-access-d8r7g\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.906992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.906950 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-metrics-certs\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.907431 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.907410 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2803a5-571f-4ead-8192-876175cb040c-openshift-service-ca\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.909277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.909254 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-metrics-certs\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.967286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.967250 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8r7g\" (UniqueName: \"kubernetes.io/projected/ec2803a5-571f-4ead-8192-876175cb040c-kube-api-access-d8r7g\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:18.988492 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:18.988402 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:19.198865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:19.198830 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:19.415172 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:19.415126 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:19.417616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:19.417593 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls\") pod \"maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:19.647674 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:19.647636 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:20.019194 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:20.019164 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp"] Nov 03 19:07:20.021563 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:07:20.021536 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2803a5_571f_4ead_8192_876175cb040c.slice/crio-490d297df64aea022de86de2eed18f0760354a2675b5fdc44a3f9e7e19018b55 WatchSource:0}: Error finding container 490d297df64aea022de86de2eed18f0760354a2675b5fdc44a3f9e7e19018b55: Status 404 returned error can't find the container with id 490d297df64aea022de86de2eed18f0760354a2675b5fdc44a3f9e7e19018b55 Nov 03 19:07:20.975266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:20.975169 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" event={"ID":"ec2803a5-571f-4ead-8192-876175cb040c","Type":"ContainerStarted","Data":"490d297df64aea022de86de2eed18f0760354a2675b5fdc44a3f9e7e19018b55"} Nov 03 19:07:21.984670 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:21.984631 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" event={"ID":"ec2803a5-571f-4ead-8192-876175cb040c","Type":"ContainerStarted","Data":"ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c"} Nov 03 19:07:21.985091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:21.984741 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:22.130336 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:22.130279 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" podStartSLOduration=3.247755256 podStartE2EDuration="4.130260934s" totalImagesPullingTime="882.505678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:22.127163464 +0000 UTC m=+1717.008579582" watchObservedRunningTime="2026-11-03 19:07:22.130260934 +0000 UTC m=+1717.011677034" Nov 03 19:07:24.196763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:24.196658 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:25.401393 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:25.401356 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:25.608569 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:25.608535 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:25.938224 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:07:25.938189 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:07:27.996221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:27.996190 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:07:28.837332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:28.837298 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:32.957572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:32.957534 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:34.518554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:34.518507 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:34.765204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:34.765164 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:34.974324 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:34.974291 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:35.204249 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:35.204183 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:35.942137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:35.942101 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:36.644080 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:36.644030 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:36.923598 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:36.923503 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:37.073430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:37.073390 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh"] Nov 03 19:07:37.073860 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:37.073615 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" containerID="cri-o://2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e" gracePeriod=30 Nov 03 19:07:40.174383 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:40.174320 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:40.619967 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:40.619922 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:40.852211 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:40.852174 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:41.103740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:41.103705 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:41.615484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:41.615451 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:07:41.615981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:41.615512 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:07:42.350529 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.350460 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 19:07:42.524764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.524725 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-podinfo\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.524960 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.524909 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-token\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-token\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525024 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.524974 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-credential-socket\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.525078 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-socket\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525154 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.525139 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istiod-ca-cert\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.525205 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-kube-api-access-gqz7k\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-kube-api-access-gqz7k\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525270 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.525253 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-data\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.525373 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-certs\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.525449 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-envoy\") pod \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\" (UID: \"2377db91-edd4-4b77-b952-6c2eb8ac1f26\") " Nov 03 19:07:42.525887 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.525857 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istiod-ca-cert" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:42.526145 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.526122 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-credential-socket" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.526538 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.526507 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.526538 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.526536 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.526705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.526666 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-socket" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.526771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.526713 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-data" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.526853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.526826 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-certs" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.527297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.527262 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-podinfo" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:42.529311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.529272 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-envoy" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:42.529416 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.529310 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-token" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.529600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.529573 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-kube-api-access-gqz7k" pod "2377db91-edd4-4b77-b952-6c2eb8ac1f26" (UID: "2377db91-edd4-4b77-b952-6c2eb8ac1f26"). InnerVolumeSpecName "kube-api-access-gqz7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:42.629038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.628919 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.629038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.628958 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.629038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.628975 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqz7k\" (UniqueName: \"kubernetes.io/projected/2377db91-edd4-4b77-b952-6c2eb8ac1f26-kube-api-access-gqz7k\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.629038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.628993 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.629679 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.629008 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.629679 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.629078 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:42.629679 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:42.629093 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2377db91-edd4-4b77-b952-6c2eb8ac1f26-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:43.156339 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.156301 5988 generic.go:356] "Generic (PLEG): container finished" podID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerID="2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e" exitCode=0 Nov 03 19:07:43.156543 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.156359 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" Nov 03 19:07:43.156543 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.156385 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" event={"ID":"2377db91-edd4-4b77-b952-6c2eb8ac1f26","Type":"ContainerDied","Data":"2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e"} Nov 03 19:07:43.156543 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.156453 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh" event={"ID":"2377db91-edd4-4b77-b952-6c2eb8ac1f26","Type":"ContainerDied","Data":"441d3460533d51f939eeb21a7b940fb697d04190b5aa3cba0a285220ca0cb7b3"} Nov 03 19:07:43.156543 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.156476 5988 scope.go:122] "RemoveContainer" containerID="2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e" Nov 03 19:07:43.167737 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.167719 5988 scope.go:122] "RemoveContainer" containerID="2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e" Nov 03 19:07:43.168036 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:07:43.168017 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e\": container with ID starting with 2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e not found: ID does not exist" containerID="2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e" Nov 03 19:07:43.168104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.168048 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e"} err="failed to get container status \"2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e\": rpc error: code = NotFound desc = could not find container \"2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e\": container with ID starting with 2afad77b59a34beae70f2293c488a1a0fe8fc0094c17b41f0280381688cd866e not found: ID does not exist" Nov 03 19:07:43.362464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.362431 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh"] Nov 03 19:07:43.455343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:43.455259 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-chgfh"] Nov 03 19:07:44.005828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:44.005787 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:44.218293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:44.218253 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:45.275105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:45.275059 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" path="/var/lib/kubelet/pods/2377db91-edd4-4b77-b952-6c2eb8ac1f26/volumes" Nov 03 19:07:45.678515 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:45.678477 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:49.745743 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:49.745673 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:50.697707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:50.697150 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:51.357833 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:51.357790 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:52.462382 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:52.462346 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52"] Nov 03 19:07:52.462892 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:52.462559 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" containerID="cri-o://929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab" gracePeriod=30 Nov 03 19:07:53.838802 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:53.838760 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:54.107165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:54.107072 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:54.369678 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:54.369540 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:55.313995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:55.313924 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:55.543692 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:55.543628 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:55.942895 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:07:55.942839 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:07:56.032637 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:56.032594 5988 patch_prober.go:29] interesting pod/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:07:56.032950 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:56.032698 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:07:57.304742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.304708 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:57.752872 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.752845 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:07:57.822766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.822733 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-credential-socket\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.822963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.822779 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istiod-ca-cert\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.822963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.822812 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-podinfo\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.822963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.822882 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-token\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-token\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.822963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.822938 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-data\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.823258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823035 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-envoy\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.823258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823063 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-credential-socket" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:57.823258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823096 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-socket\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.823258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823198 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-certs\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.823473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823265 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-data" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:57.823473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823275 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istiod-ca-cert" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:07:57.823473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823324 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-kube-api-access-cgfgh\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-kube-api-access-cgfgh\") pod \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\" (UID: \"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf\") " Nov 03 19:07:57.823473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823454 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-socket" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:57.823865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823831 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-certs" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:57.824005 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.823984 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.824073 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.824009 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.824073 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.824024 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.824073 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.824038 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.824073 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.824053 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.825328 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.825306 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-token" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:57.825439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.825405 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-kube-api-access-cgfgh" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "kube-api-access-cgfgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:07:57.825439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.825419 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-podinfo" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:07:57.825439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.825417 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-envoy" pod "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" (UID: "93ca1f38-4563-4ce9-b0fa-77c12e34b8bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:07:57.925488 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.925404 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.925488 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.925433 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.925488 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.925442 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:57.925488 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:57.925452 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfgh\" (UniqueName: \"kubernetes.io/projected/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf-kube-api-access-cgfgh\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:07:58.284722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.284601 5988 generic.go:356] "Generic (PLEG): container finished" podID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerID="929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab" exitCode=0 Nov 03 19:07:58.284722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.284655 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" Nov 03 19:07:58.284722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.284677 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" event={"ID":"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf","Type":"ContainerDied","Data":"929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab"} Nov 03 19:07:58.284947 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.284771 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52" event={"ID":"93ca1f38-4563-4ce9-b0fa-77c12e34b8bf","Type":"ContainerDied","Data":"6f7983d7fe3ecb97ee5e794c75082c2842cc38cf993f632ee55899d1d51c4ed4"} Nov 03 19:07:58.284947 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.284796 5988 scope.go:122] "RemoveContainer" containerID="929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab" Nov 03 19:07:58.297377 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.297349 5988 scope.go:122] "RemoveContainer" containerID="d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4" Nov 03 19:07:58.307537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.307507 5988 scope.go:122] "RemoveContainer" containerID="929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab" Nov 03 19:07:58.307900 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:07:58.307872 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab\": container with ID starting with 929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab not found: ID does not exist" containerID="929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab" Nov 03 19:07:58.307945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.307903 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab"} err="failed to get container status \"929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab\": rpc error: code = NotFound desc = could not find container \"929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab\": container with ID starting with 929375ed7ed53e5d08469dd38300a89844a8cb0f5d7f44f8742b94a3902d86ab not found: ID does not exist" Nov 03 19:07:58.307945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.307923 5988 scope.go:122] "RemoveContainer" containerID="d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4" Nov 03 19:07:58.308219 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:07:58.308200 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4\": container with ID starting with d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4 not found: ID does not exist" containerID="d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4" Nov 03 19:07:58.308261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.308227 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4"} err="failed to get container status \"d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4\": rpc error: code = NotFound desc = could not find container \"d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4\": container with ID starting with d7d58cb1c9c5592e63c811c210e7c5e9d0649f24dd2e1c4f6282b4f4e60f81a4 not found: ID does not exist" Nov 03 19:07:58.492503 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.492467 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:07:58.506626 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.506588 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52"] Nov 03 19:07:58.561338 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.561261 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6765e083-openshift-default-8486d6c5fc-5df52"] Nov 03 19:07:58.732307 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.732274 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:58.978901 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:58.978853 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:07:59.273915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:07:59.273834 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" path="/var/lib/kubelet/pods/93ca1f38-4563-4ce9-b0fa-77c12e34b8bf/volumes" Nov 03 19:08:01.894256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:01.894219 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:02.127667 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:02.127612 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:03.187309 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:03.187265 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:03.397146 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:03.397099 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:03.902098 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:03.902058 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:04.268890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:04.268612 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:07.691570 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:07.691531 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn"] Nov 03 19:08:07.692205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:07.692151 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" containerID="cri-o://043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5" gracePeriod=30 Nov 03 19:08:07.738333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:07.738274 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp"] Nov 03 19:08:07.738542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:07.738517 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" containerID="cri-o://b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737" gracePeriod=30 Nov 03 19:08:08.442819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:08.442781 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:09.831438 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:09.831403 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:10.052949 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:10.052915 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:11.102731 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:11.102661 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:11.532055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:11.531976 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:12.173138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:12.173059 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:12.438500 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:12.438402 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:12.876242 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:12.876204 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 19:08:12.876453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:12.876272 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 19:08:13.030161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.030127 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:08:13.076142 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.076115 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:08:13.142979 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.142942 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-credential-socket\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.142979 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.142984 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-kube-api-access-54t8s\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-kube-api-access-54t8s\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.143256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143019 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-envoy\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.143256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143078 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-certs\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.143256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143113 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-data\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.143256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143211 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-credential-socket\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.143256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143252 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-kube-api-access-7jq4s\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-kube-api-access-7jq4s\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.143507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143302 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-socket\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.143507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143319 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-credential-socket" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.143507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143402 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-podinfo\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.143507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143478 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-token\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-token\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.143721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143513 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475f7e65-3be6-446b-9be5-86d850b81602-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/475f7e65-3be6-446b-9be5-86d850b81602-istiod-ca-cert\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.143721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143549 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-socket" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.143721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143605 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-data\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.143721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143675 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-envoy\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.143721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143675 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-credential-socket" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.144062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143771 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54416f7b-5912-4ae0-a003-4cd0addb5113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54416f7b-5912-4ae0-a003-4cd0addb5113-istiod-ca-cert\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.144062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143872 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/475f7e65-3be6-446b-9be5-86d850b81602-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/475f7e65-3be6-446b-9be5-86d850b81602-istio-podinfo\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.144062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143913 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-istio-token\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-istio-token\") pod \"475f7e65-3be6-446b-9be5-86d850b81602\" (UID: \"475f7e65-3be6-446b-9be5-86d850b81602\") " Nov 03 19:08:13.144062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143943 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-certs\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.144062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.143970 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-socket\") pod \"54416f7b-5912-4ae0-a003-4cd0addb5113\" (UID: \"54416f7b-5912-4ae0-a003-4cd0addb5113\") " Nov 03 19:08:13.145441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.144453 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-certs" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.145441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.144868 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-data" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.145441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.144955 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.145441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.144979 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.145441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.144995 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.145441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.145012 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.145441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.145171 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475f7e65-3be6-446b-9be5-86d850b81602-istiod-ca-cert" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:13.146017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.145483 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-data" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.146017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.145621 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-socket" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.146017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.145907 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-envoy" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.146184 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.146084 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-kube-api-access-54t8s" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "kube-api-access-54t8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:13.146337 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.146312 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-kube-api-access-7jq4s" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "kube-api-access-7jq4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:13.146436 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.146414 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-certs" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.146582 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.146563 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54416f7b-5912-4ae0-a003-4cd0addb5113-istiod-ca-cert" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:08:13.146650 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.146562 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-envoy" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:08:13.147133 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.147101 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-podinfo" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:13.147570 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.147546 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/475f7e65-3be6-446b-9be5-86d850b81602-istio-podinfo" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:08:13.147660 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.147589 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-token" pod "54416f7b-5912-4ae0-a003-4cd0addb5113" (UID: "54416f7b-5912-4ae0-a003-4cd0addb5113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:13.147964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.147944 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-istio-token" pod "475f7e65-3be6-446b-9be5-86d850b81602" (UID: "475f7e65-3be6-446b-9be5-86d850b81602"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246096 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246128 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54416f7b-5912-4ae0-a003-4cd0addb5113-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246138 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/475f7e65-3be6-446b-9be5-86d850b81602-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246147 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246156 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246167 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246180 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54t8s\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-kube-api-access-54t8s\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246189 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246197 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.246196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246206 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jq4s\" (UniqueName: \"kubernetes.io/projected/475f7e65-3be6-446b-9be5-86d850b81602-kube-api-access-7jq4s\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.247193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246215 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.247193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246223 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54416f7b-5912-4ae0-a003-4cd0addb5113-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.247193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246233 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/475f7e65-3be6-446b-9be5-86d850b81602-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.247193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.246245 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/475f7e65-3be6-446b-9be5-86d850b81602-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:08:13.366336 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.366298 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:13.398087 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.398052 5988 generic.go:356] "Generic (PLEG): container finished" podID="475f7e65-3be6-446b-9be5-86d850b81602" containerID="043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5" exitCode=0 Nov 03 19:08:13.398266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.398101 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" Nov 03 19:08:13.398266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.398137 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" event={"ID":"475f7e65-3be6-446b-9be5-86d850b81602","Type":"ContainerDied","Data":"043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5"} Nov 03 19:08:13.398266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.398201 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn" event={"ID":"475f7e65-3be6-446b-9be5-86d850b81602","Type":"ContainerDied","Data":"7f8a3a021cd8214b9a81cd6188628ab1732c00b894969871bceef93eef7e4b4a"} Nov 03 19:08:13.398266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.398232 5988 scope.go:122] "RemoveContainer" containerID="043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5" Nov 03 19:08:13.399770 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.399746 5988 generic.go:356] "Generic (PLEG): container finished" podID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerID="b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737" exitCode=0 Nov 03 19:08:13.399906 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.399892 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" event={"ID":"54416f7b-5912-4ae0-a003-4cd0addb5113","Type":"ContainerDied","Data":"b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737"} Nov 03 19:08:13.399970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.399927 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" event={"ID":"54416f7b-5912-4ae0-a003-4cd0addb5113","Type":"ContainerDied","Data":"95050668c994e888a0fa6bfded3729c42670a8109390b952b0e0db1b0a874c40"} Nov 03 19:08:13.400031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.399982 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp" Nov 03 19:08:13.415289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.415265 5988 scope.go:122] "RemoveContainer" containerID="043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5" Nov 03 19:08:13.415628 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:08:13.415611 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5\": container with ID starting with 043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5 not found: ID does not exist" containerID="043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5" Nov 03 19:08:13.415754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.415638 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5"} err="failed to get container status \"043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5\": rpc error: code = NotFound desc = could not find container \"043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5\": container with ID starting with 043fa0cc9eb779b4b2553d05e302573d02d8f2d3a244d93ee0bca349cbedb8e5 not found: ID does not exist" Nov 03 19:08:13.415754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.415656 5988 scope.go:122] "RemoveContainer" containerID="b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737" Nov 03 19:08:13.432052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.432025 5988 scope.go:122] "RemoveContainer" containerID="b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737" Nov 03 19:08:13.432396 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:08:13.432374 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737\": container with ID starting with b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737 not found: ID does not exist" containerID="b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737" Nov 03 19:08:13.432470 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.432405 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737"} err="failed to get container status \"b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737\": rpc error: code = NotFound desc = could not find container \"b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737\": container with ID starting with b297c70dd1eeccd951ede6995c77365e231151b9747568c7fd540985cebb0737 not found: ID does not exist" Nov 03 19:08:13.557629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.557536 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn"] Nov 03 19:08:13.602365 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.602330 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-jk6qn"] Nov 03 19:08:13.665800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.665761 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp"] Nov 03 19:08:13.709549 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.709504 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-64gzp"] Nov 03 19:08:13.805638 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:13.805593 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:14.635422 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:14.635386 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:14.844404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:14.844355 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:15.280705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:15.280112 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475f7e65-3be6-446b-9be5-86d850b81602" path="/var/lib/kubelet/pods/475f7e65-3be6-446b-9be5-86d850b81602/volumes" Nov 03 19:08:15.281001 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:15.280815 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" path="/var/lib/kubelet/pods/54416f7b-5912-4ae0-a003-4cd0addb5113/volumes" Nov 03 19:08:16.993198 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:16.993160 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:17.235717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:17.235668 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:17.493000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:17.492967 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:19.468570 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:19.468530 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:19.871391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:19.871348 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:20.433487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:20.433453 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:20.686485 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:20.686398 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:25.323708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:25.316193 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:25.938865 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:08:25.938825 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:08:31.019064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:31.019024 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:34.163905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:34.163869 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:35.028856 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:35.028816 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:35.241530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:35.241499 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:35.637620 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:35.637588 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:35.980333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:35.980241 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:41.027903 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:41.026312 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:41.866626 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:41.866590 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:45.296803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.296766 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr"] Nov 03 19:08:45.297511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297494 5988 cpu_manager.go:414] "containerMap: removing container" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" Nov 03 19:08:45.297572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297511 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" Nov 03 19:08:45.297572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297551 5988 cpu_manager.go:414] "containerMap: removing container" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" Nov 03 19:08:45.297572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297560 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297577 5988 cpu_manager.go:414] "containerMap: removing container" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297582 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297600 5988 cpu_manager.go:414] "containerMap: removing container" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297605 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297618 5988 cpu_manager.go:414] "containerMap: removing container" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297622 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297734 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" Nov 03 19:08:45.297744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297745 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="2377db91-edd4-4b77-b952-6c2eb8ac1f26" containerName="istio-proxy" Nov 03 19:08:45.298124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297756 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="475f7e65-3be6-446b-9be5-86d850b81602" containerName="istio-proxy" Nov 03 19:08:45.298124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297766 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="54416f7b-5912-4ae0-a003-4cd0addb5113" containerName="istio-proxy" Nov 03 19:08:45.298124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.297773 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ca1f38-4563-4ce9-b0fa-77c12e34b8bf" containerName="istio-proxy" Nov 03 19:08:45.300824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.300804 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.352860 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.350071 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-bf3e1c-serving-cert\"" Nov 03 19:08:45.400718 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.400668 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr"] Nov 03 19:08:45.408589 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.408553 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.408814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.408792 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.408878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.408830 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndzhk\" (UniqueName: \"kubernetes.io/projected/6f095571-f9ab-4d4e-8dcb-72b49195e131-kube-api-access-ndzhk\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.408957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.408935 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f095571-f9ab-4d4e-8dcb-72b49195e131-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.510015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.509983 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.510231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.510031 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndzhk\" (UniqueName: \"kubernetes.io/projected/6f095571-f9ab-4d4e-8dcb-72b49195e131-kube-api-access-ndzhk\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.510231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.510102 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f095571-f9ab-4d4e-8dcb-72b49195e131-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.510231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.510205 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.510428 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:08:45.510409 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-serving-cert: secret "maas-api-e2e-shared-a-w1-bf3e1c-serving-cert" not found Nov 03 19:08:45.510494 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:08:45.510480 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls podName:6f095571-f9ab-4d4e-8dcb-72b49195e131 nodeName:}" failed. No retries permitted until 2026-11-03 19:08:46.010461233 +0000 UTC m=+1800.891877317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls") pod "maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" (UID: "6f095571-f9ab-4d4e-8dcb-72b49195e131") : secret "maas-api-e2e-shared-a-w1-bf3e1c-serving-cert" not found Nov 03 19:08:45.510860 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.510837 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f095571-f9ab-4d4e-8dcb-72b49195e131-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.512775 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.512756 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.562722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.562591 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:45.568357 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.568328 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndzhk\" (UniqueName: \"kubernetes.io/projected/6f095571-f9ab-4d4e-8dcb-72b49195e131-kube-api-access-ndzhk\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:45.676165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.676134 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:08:45.798751 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:45.798718 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:46.018592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:46.018559 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:46.020967 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:46.020945 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:46.214313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:46.214237 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:46.591981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:46.591956 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr"] Nov 03 19:08:46.594434 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:08:46.594408 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f095571_f9ab_4d4e_8dcb_72b49195e131.slice/crio-a97c9b444ce75d5732eff8ce42321494865ca746b03243cdc685f5ecb1d54111 WatchSource:0}: Error finding container a97c9b444ce75d5732eff8ce42321494865ca746b03243cdc685f5ecb1d54111: Status 404 returned error can't find the container with id a97c9b444ce75d5732eff8ce42321494865ca746b03243cdc685f5ecb1d54111 Nov 03 19:08:46.685773 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:46.685735 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" event={"ID":"6f095571-f9ab-4d4e-8dcb-72b49195e131","Type":"ContainerStarted","Data":"a97c9b444ce75d5732eff8ce42321494865ca746b03243cdc685f5ecb1d54111"} Nov 03 19:08:47.704659 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:47.700028 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" event={"ID":"6f095571-f9ab-4d4e-8dcb-72b49195e131","Type":"ContainerStarted","Data":"6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e"} Nov 03 19:08:47.704659 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:47.700357 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:51.053319 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:51.053279 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:51.275010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:51.274977 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:53.733776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:53.733742 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:08:53.845544 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:53.845449 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" podStartSLOduration=8.050195675 podStartE2EDuration="8.845436322s" totalImagesPullingTime="795.240647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:47.816559571 +0000 UTC m=+1802.697975674" watchObservedRunningTime="2026-11-03 19:08:53.845436322 +0000 UTC m=+1808.726852424" Nov 03 19:08:55.708966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:55.708905 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:08:55.941137 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:08:55.941088 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:08:58.699751 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:08:58.699661 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:00.256741 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:00.256702 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:04.799738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:04.797978 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:05.018611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:05.018558 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:05.224133 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:05.224084 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:07.015014 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:07.014967 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:08.657437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:08.657407 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:09.428321 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:09.428285 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:09.960012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:09.959977 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:10.388707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:10.387004 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:14.038519 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:14.038471 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:14.337234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:14.337199 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:15.459552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:15.459488 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:17.366745 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:17.354281 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:17.581708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:17.575369 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:20.460749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:20.460711 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:20.664740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:20.664695 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:25.708535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:25.708499 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:25.940152 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:09:25.940103 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:09:30.304971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:30.304900 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:32.816203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:32.816169 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:09:34.759195 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:34.759157 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:34.975355 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:34.975297 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:35.184983 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:35.184942 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:39.491048 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:39.490956 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:44.030337 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:44.030261 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:48.668757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:48.668673 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:51.461478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:51.461439 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:53.133560 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:53.133506 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:54.886533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:54.886496 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:09:55.938452 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:09:55.938379 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:09:57.734267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:09:57.734223 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:01.078488 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:01.078439 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:02.466524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:02.466459 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:07.409858 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:07.409808 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:08.992351 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:08.992298 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:09.113511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:09.113477 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9"] Nov 03 19:10:09.113764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:09.113735 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" containerID="cri-o://75725ac282a1982e8b67b85ae7009036f15f0fad15cd646ebbe49c58b18dad68" gracePeriod=30 Nov 03 19:10:09.415982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:09.415924 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:11.166162 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:11.166121 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:11.395355 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:11.395283 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:12.836704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:12.836076 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:14.361317 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.361286 5988 generic.go:356] "Generic (PLEG): container finished" podID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerID="75725ac282a1982e8b67b85ae7009036f15f0fad15cd646ebbe49c58b18dad68" exitCode=0 Nov 03 19:10:14.361743 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.361349 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" event={"ID":"023d7eaf-9220-48e0-ab41-3a09a7de82e3","Type":"ContainerDied","Data":"75725ac282a1982e8b67b85ae7009036f15f0fad15cd646ebbe49c58b18dad68"} Nov 03 19:10:14.406657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.406627 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:10:14.595178 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595136 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-podinfo\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595318 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-kube-api-access-9vl88\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-kube-api-access-9vl88\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595374 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-envoy\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595402 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-token\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-token\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595470 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-credential-socket\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595546 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istiod-ca-cert\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595720 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595619 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-certs\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595786 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595765 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-data\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.595860 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595842 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-socket\") pod \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\" (UID: \"023d7eaf-9220-48e0-ab41-3a09a7de82e3\") " Nov 03 19:10:14.596074 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.595960 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-credential-socket" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:14.596203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596088 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-certs" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:14.596203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596181 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-data" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:14.596340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596264 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istiod-ca-cert" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:10:14.596726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596618 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-socket" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:14.596814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596735 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.596814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596750 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.596814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596758 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.596814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596767 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.596814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.596776 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.598023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.597995 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-podinfo" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:10:14.598720 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.598653 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-kube-api-access-9vl88" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "kube-api-access-9vl88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:10:14.598877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.598838 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-token" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:10:14.599576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.599553 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-envoy" pod "023d7eaf-9220-48e0-ab41-3a09a7de82e3" (UID: "023d7eaf-9220-48e0-ab41-3a09a7de82e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:14.698088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.697992 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vl88\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-kube-api-access-9vl88\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.698088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.698025 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.698088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.698035 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:14.698088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:14.698043 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023d7eaf-9220-48e0-ab41-3a09a7de82e3-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:15.369028 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:15.368998 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" Nov 03 19:10:15.369509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:15.369034 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9" event={"ID":"023d7eaf-9220-48e0-ab41-3a09a7de82e3","Type":"ContainerDied","Data":"3e94197ae0bd7054a67e846c798933e1760f1381c5821d46250a9b62e29c1dda"} Nov 03 19:10:15.369509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:15.369113 5988 scope.go:122] "RemoveContainer" containerID="75725ac282a1982e8b67b85ae7009036f15f0fad15cd646ebbe49c58b18dad68" Nov 03 19:10:15.572170 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:15.572130 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9"] Nov 03 19:10:15.618849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:15.618815 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-t4rd9"] Nov 03 19:10:16.096082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:16.096048 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:16.317894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:16.317855 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:17.273990 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:17.273957 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" path="/var/lib/kubelet/pods/023d7eaf-9220-48e0-ab41-3a09a7de82e3/volumes" Nov 03 19:10:17.809865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:17.809793 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:18.454771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:18.454731 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:18.674480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:18.674439 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:22.221703 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:22.219219 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:22.425601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:22.425554 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:22.816711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:22.773345 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:24.358405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:24.358328 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99"] Nov 03 19:10:24.358889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:24.358523 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" containerID="cri-o://d5610ecb38181177136e9185de4195842a43f33aa247a5b0728bba5cfabe2f2e" gracePeriod=30 Nov 03 19:10:25.936436 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:10:25.936393 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:10:27.425518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:27.425478 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:29.007618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.007583 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:10:29.008035 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.007641 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:10:29.484578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.484545 5988 generic.go:356] "Generic (PLEG): container finished" podID="a43e5825-60f0-4768-be89-bd2136c65a57" containerID="d5610ecb38181177136e9185de4195842a43f33aa247a5b0728bba5cfabe2f2e" exitCode=0 Nov 03 19:10:29.484725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.484613 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" event={"ID":"a43e5825-60f0-4768-be89-bd2136c65a57","Type":"ContainerDied","Data":"d5610ecb38181177136e9185de4195842a43f33aa247a5b0728bba5cfabe2f2e"} Nov 03 19:10:29.636833 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.636773 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:10:29.763915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.763880 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43e5825-60f0-4768-be89-bd2136c65a57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43e5825-60f0-4768-be89-bd2136c65a57-istiod-ca-cert\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.763927 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-istio-token\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-istio-token\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.763966 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-credential-socket\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764020 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-data\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764224 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764130 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-kube-api-access-ppcw6\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-kube-api-access-ppcw6\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764224 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764200 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-socket\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764231 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-certs\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764322 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a43e5825-60f0-4768-be89-bd2136c65a57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43e5825-60f0-4768-be89-bd2136c65a57-istio-podinfo\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764330 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-credential-socket" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:29.764437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764386 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-data" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:29.764437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764418 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43e5825-60f0-4768-be89-bd2136c65a57-istiod-ca-cert" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:10:29.764602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764453 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-envoy\") pod \"a43e5825-60f0-4768-be89-bd2136c65a57\" (UID: \"a43e5825-60f0-4768-be89-bd2136c65a57\") " Nov 03 19:10:29.764602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764550 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-socket" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:29.764746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.764672 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-certs" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:29.765295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.765266 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43e5825-60f0-4768-be89-bd2136c65a57-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.765295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.765299 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.765501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.765316 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.765501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.765331 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.765501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.765345 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.766331 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.766302 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-istio-token" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:10:29.766514 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.766490 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-kube-api-access-ppcw6" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "kube-api-access-ppcw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:10:29.766580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.766519 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-envoy" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:10:29.766580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.766557 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a43e5825-60f0-4768-be89-bd2136c65a57-istio-podinfo" pod "a43e5825-60f0-4768-be89-bd2136c65a57" (UID: "a43e5825-60f0-4768-be89-bd2136c65a57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:10:29.866917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.866884 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcw6\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-kube-api-access-ppcw6\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.866917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.866912 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43e5825-60f0-4768-be89-bd2136c65a57-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.866917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.866922 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43e5825-60f0-4768-be89-bd2136c65a57-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:29.867147 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:29.866930 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43e5825-60f0-4768-be89-bd2136c65a57-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:10:30.492103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:30.492075 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" Nov 03 19:10:30.492103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:30.492089 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99" event={"ID":"a43e5825-60f0-4768-be89-bd2136c65a57","Type":"ContainerDied","Data":"24eec94535732dd03754cea81b174d7e8548eb088395d906ee50e828c1932896"} Nov 03 19:10:30.492578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:30.492143 5988 scope.go:122] "RemoveContainer" containerID="d5610ecb38181177136e9185de4195842a43f33aa247a5b0728bba5cfabe2f2e" Nov 03 19:10:30.719306 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:30.719269 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99"] Nov 03 19:10:30.766098 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:30.766026 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2mq99"] Nov 03 19:10:31.273719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:31.273665 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" path="/var/lib/kubelet/pods/a43e5825-60f0-4768-be89-bd2136c65a57/volumes" Nov 03 19:10:31.868996 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:31.868945 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:32.305512 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:32.305363 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:32.515406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:32.515365 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:32.720405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:32.720370 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:32.922773 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:32.922730 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:33.220781 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:33.220739 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:37.587899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:37.587866 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:37.790233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:37.790195 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:38.010278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:38.010196 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:40.186675 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:40.186633 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:41.080749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:41.080710 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:41.294066 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:41.294035 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:41.664515 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:41.664475 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:10:42.881165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:42.881111 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:48.606208 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:48.606123 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:51.823090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:51.823060 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:52.619360 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:52.619314 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:53.384166 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:53.384127 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:53.612642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:53.612596 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:54.199711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:54.199618 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:54.518038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:54.517942 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:55.937364 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:10:55.937321 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:10:58.100490 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:58.100459 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:58.354789 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:58.354674 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:10:58.564992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:10:58.564941 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:03.341535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:03.341464 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:04.046342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.046295 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:04.250893 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.250853 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:04.741406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.741376 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs"] Nov 03 19:11:04.742182 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.742130 5988 cpu_manager.go:414] "containerMap: removing container" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" Nov 03 19:11:04.742340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.742319 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" Nov 03 19:11:04.742459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.742353 5988 cpu_manager.go:414] "containerMap: removing container" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" Nov 03 19:11:04.742459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.742359 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" Nov 03 19:11:04.742578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.742512 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="023d7eaf-9220-48e0-ab41-3a09a7de82e3" containerName="istio-proxy" Nov 03 19:11:04.742578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.742535 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43e5825-60f0-4768-be89-bd2136c65a57" containerName="istio-proxy" Nov 03 19:11:04.748219 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.748188 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:04.799548 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.799510 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs"] Nov 03 19:11:04.842561 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.842527 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-efd81c-dockercfg-w2l66\"" Nov 03 19:11:04.852125 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.852102 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-efd81c\"" Nov 03 19:11:04.913224 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.913190 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330f5ff2-913a-4a92-bc49-7204c9480910-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:04.913391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:04.913327 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshx2\" (UniqueName: \"kubernetes.io/projected/330f5ff2-913a-4a92-bc49-7204c9480910-kube-api-access-lshx2\") pod \"payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:05.014971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.014897 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330f5ff2-913a-4a92-bc49-7204c9480910-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:05.015118 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.014981 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshx2\" (UniqueName: \"kubernetes.io/projected/330f5ff2-913a-4a92-bc49-7204c9480910-kube-api-access-lshx2\") pod \"payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:05.015742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.015721 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330f5ff2-913a-4a92-bc49-7204c9480910-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:05.073004 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.072972 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshx2\" (UniqueName: \"kubernetes.io/projected/330f5ff2-913a-4a92-bc49-7204c9480910-kube-api-access-lshx2\") pod \"payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:05.365072 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.365042 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:05.676830 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.676746 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:05.735186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.735162 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs"] Nov 03 19:11:05.736562 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:11:05.736534 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330f5ff2_913a_4a92_bc49_7204c9480910.slice/crio-68969c12f331c4241c65bfcdedae270be0c608ce0557fddb08265808fe3c8356 WatchSource:0}: Error finding container 68969c12f331c4241c65bfcdedae270be0c608ce0557fddb08265808fe3c8356: Status 404 returned error can't find the container with id 68969c12f331c4241c65bfcdedae270be0c608ce0557fddb08265808fe3c8356 Nov 03 19:11:05.784913 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:05.784877 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" event={"ID":"330f5ff2-913a-4a92-bc49-7204c9480910","Type":"ContainerStarted","Data":"68969c12f331c4241c65bfcdedae270be0c608ce0557fddb08265808fe3c8356"} Nov 03 19:11:06.793828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:06.793792 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" event={"ID":"330f5ff2-913a-4a92-bc49-7204c9480910","Type":"ContainerStarted","Data":"2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7"} Nov 03 19:11:06.794245 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:06.793881 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:06.907013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:06.906948 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" podStartSLOduration=2.9069292239999998 podStartE2EDuration="2.906929224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:06.903928439 +0000 UTC m=+1941.785344556" watchObservedRunningTime="2026-11-03 19:11:06.906929224 +0000 UTC m=+1941.788345327" Nov 03 19:11:07.056249 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:07.056154 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:07.776247 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:07.776210 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:08.138619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.138582 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp"] Nov 03 19:11:08.139509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.139474 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" podUID="ec2803a5-571f-4ead-8192-876175cb040c" containerName="maas-api" containerID="cri-o://ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c" gracePeriod=30 Nov 03 19:11:08.225221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.225189 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:08.435317 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.435198 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:08.470819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.470789 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:11:08.658334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.658293 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls\") pod \"ec2803a5-571f-4ead-8192-876175cb040c\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " Nov 03 19:11:08.658537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.658413 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2803a5-571f-4ead-8192-876175cb040c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2803a5-571f-4ead-8192-876175cb040c-openshift-service-ca\") pod \"ec2803a5-571f-4ead-8192-876175cb040c\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " Nov 03 19:11:08.658537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.658466 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-metrics-certs\") pod \"ec2803a5-571f-4ead-8192-876175cb040c\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " Nov 03 19:11:08.658785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.658538 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2803a5-571f-4ead-8192-876175cb040c-kube-api-access-d8r7g\" (UniqueName: \"kubernetes.io/projected/ec2803a5-571f-4ead-8192-876175cb040c-kube-api-access-d8r7g\") pod \"ec2803a5-571f-4ead-8192-876175cb040c\" (UID: \"ec2803a5-571f-4ead-8192-876175cb040c\") " Nov 03 19:11:08.659951 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.659921 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2803a5-571f-4ead-8192-876175cb040c-openshift-service-ca" pod "ec2803a5-571f-4ead-8192-876175cb040c" (UID: "ec2803a5-571f-4ead-8192-876175cb040c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:11:08.661838 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.661799 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls" pod "ec2803a5-571f-4ead-8192-876175cb040c" (UID: "ec2803a5-571f-4ead-8192-876175cb040c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:11:08.662416 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.662371 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2803a5-571f-4ead-8192-876175cb040c-kube-api-access-d8r7g" pod "ec2803a5-571f-4ead-8192-876175cb040c" (UID: "ec2803a5-571f-4ead-8192-876175cb040c"). InnerVolumeSpecName "kube-api-access-d8r7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:11:08.664778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.662670 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-metrics-certs" pod "ec2803a5-571f-4ead-8192-876175cb040c" (UID: "ec2803a5-571f-4ead-8192-876175cb040c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:11:08.761031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.760940 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:08.761031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.760982 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2803a5-571f-4ead-8192-876175cb040c-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:08.761031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.760995 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2803a5-571f-4ead-8192-876175cb040c-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:08.761031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.761007 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8r7g\" (UniqueName: \"kubernetes.io/projected/ec2803a5-571f-4ead-8192-876175cb040c-kube-api-access-d8r7g\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:08.816566 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.816529 5988 generic.go:356] "Generic (PLEG): container finished" podID="ec2803a5-571f-4ead-8192-876175cb040c" containerID="ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c" exitCode=0 Nov 03 19:11:08.816785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.816590 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" Nov 03 19:11:08.816785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.816624 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" event={"ID":"ec2803a5-571f-4ead-8192-876175cb040c","Type":"ContainerDied","Data":"ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c"} Nov 03 19:11:08.816785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.816727 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp" event={"ID":"ec2803a5-571f-4ead-8192-876175cb040c","Type":"ContainerDied","Data":"490d297df64aea022de86de2eed18f0760354a2675b5fdc44a3f9e7e19018b55"} Nov 03 19:11:08.816785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.816758 5988 scope.go:122] "RemoveContainer" containerID="ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c" Nov 03 19:11:08.830091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.830067 5988 scope.go:122] "RemoveContainer" containerID="ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c" Nov 03 19:11:08.830369 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:11:08.830347 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c\": container with ID starting with ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c not found: ID does not exist" containerID="ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c" Nov 03 19:11:08.830444 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.830382 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c"} err="failed to get container status \"ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c\": rpc error: code = NotFound desc = could not find container \"ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c\": container with ID starting with ff3fb85382836e4fe1cddcd191793cc55d2e4cbae2b3535a51511d073a27295c not found: ID does not exist" Nov 03 19:11:08.972511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:08.972477 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp"] Nov 03 19:11:09.019278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:09.019187 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ac14ff6-5877657754-vv4hp"] Nov 03 19:11:09.308409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:09.308312 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2803a5-571f-4ead-8192-876175cb040c" path="/var/lib/kubelet/pods/ec2803a5-571f-4ead-8192-876175cb040c/volumes" Nov 03 19:11:10.046280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:10.046233 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:10.307254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:10.307137 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:10.514736 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:10.514674 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:11.031302 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:11.031256 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:13.070503 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:13.070460 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:13.284612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:13.284573 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:13.518496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:13.518416 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:15.406678 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:15.406634 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:16.673008 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:16.672967 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:16.911245 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:16.911194 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:17.196299 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.196265 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t"] Nov 03 19:11:17.200704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.197457 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ec2803a5-571f-4ead-8192-876175cb040c" containerName="maas-api" Nov 03 19:11:17.201849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.201825 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2803a5-571f-4ead-8192-876175cb040c" containerName="maas-api" Nov 03 19:11:17.215709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.202230 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2803a5-571f-4ead-8192-876175cb040c" containerName="maas-api" Nov 03 19:11:17.225469 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.225438 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.282967 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.282944 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t"] Nov 03 19:11:17.314213 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.314181 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-042f66\"" Nov 03 19:11:17.314213 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.314202 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-042f66-dockercfg-5n6pq\"" Nov 03 19:11:17.344891 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.344858 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f"] Nov 03 19:11:17.347124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.347094 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tjcr\" (UniqueName: \"kubernetes.io/projected/aae6d193-38ce-41a5-9c1e-27a7996533ce-kube-api-access-6tjcr\") pod \"payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.347333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.347303 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae6d193-38ce-41a5-9c1e-27a7996533ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.349716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.349695 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.395223 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.395029 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f"] Nov 03 19:11:17.448645 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.448574 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tjcr\" (UniqueName: \"kubernetes.io/projected/aae6d193-38ce-41a5-9c1e-27a7996533ce-kube-api-access-6tjcr\") pod \"payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.448812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.448726 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae6d193-38ce-41a5-9c1e-27a7996533ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.448863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.448843 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66gsp\" (UniqueName: \"kubernetes.io/projected/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-kube-api-access-66gsp\") pod \"payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.448999 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.448975 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.449461 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.449441 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae6d193-38ce-41a5-9c1e-27a7996533ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.506313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.506280 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tjcr\" (UniqueName: \"kubernetes.io/projected/aae6d193-38ce-41a5-9c1e-27a7996533ce-kube-api-access-6tjcr\") pod \"payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.550931 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.550897 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66gsp\" (UniqueName: \"kubernetes.io/projected/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-kube-api-access-66gsp\") pod \"payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.551061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.551003 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.551660 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.551641 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.578532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.578504 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:17.612480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.612443 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66gsp\" (UniqueName: \"kubernetes.io/projected/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-kube-api-access-66gsp\") pod \"payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.666577 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.666545 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:17.813322 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.813290 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:11:17.952546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.952507 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t"] Nov 03 19:11:17.954449 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:11:17.954409 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae6d193_38ce_41a5_9c1e_27a7996533ce.slice/crio-860bdeec3682e6a736fe547a0a9589960163ed29901cae1cd6097913fcf9b002 WatchSource:0}: Error finding container 860bdeec3682e6a736fe547a0a9589960163ed29901cae1cd6097913fcf9b002: Status 404 returned error can't find the container with id 860bdeec3682e6a736fe547a0a9589960163ed29901cae1cd6097913fcf9b002 Nov 03 19:11:17.979070 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:17.979039 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" event={"ID":"aae6d193-38ce-41a5-9c1e-27a7996533ce","Type":"ContainerStarted","Data":"860bdeec3682e6a736fe547a0a9589960163ed29901cae1cd6097913fcf9b002"} Nov 03 19:11:18.061985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.061952 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f"] Nov 03 19:11:18.063027 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:11:18.062988 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c44a606_8d5c_4f14_b77d_b5c4c0ca2c40.slice/crio-6dd152b15baf3a60625114ac94f4f793f1a5f528ae37fcd71fec404fe7b908d4 WatchSource:0}: Error finding container 6dd152b15baf3a60625114ac94f4f793f1a5f528ae37fcd71fec404fe7b908d4: Status 404 returned error can't find the container with id 6dd152b15baf3a60625114ac94f4f793f1a5f528ae37fcd71fec404fe7b908d4 Nov 03 19:11:18.385301 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.385269 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:18.593749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.593670 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:18.877839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.875358 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:18.987166 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.987127 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" event={"ID":"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40","Type":"ContainerStarted","Data":"e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b"} Nov 03 19:11:18.987371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.987196 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" event={"ID":"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40","Type":"ContainerStarted","Data":"6dd152b15baf3a60625114ac94f4f793f1a5f528ae37fcd71fec404fe7b908d4"} Nov 03 19:11:18.987371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.987225 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:18.989295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.989270 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" event={"ID":"aae6d193-38ce-41a5-9c1e-27a7996533ce","Type":"ContainerStarted","Data":"8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266"} Nov 03 19:11:18.989425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:18.989316 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:19.099163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:19.099100 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" podStartSLOduration=2.099083176 podStartE2EDuration="2.099083176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:19.097732984 +0000 UTC m=+1953.979149083" watchObservedRunningTime="2026-11-03 19:11:19.099083176 +0000 UTC m=+1953.980499280" Nov 03 19:11:19.208936 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:19.207417 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" podStartSLOduration=2.207398734 podStartE2EDuration="2.207398734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:19.206206835 +0000 UTC m=+1954.087622937" watchObservedRunningTime="2026-11-03 19:11:19.207398734 +0000 UTC m=+1954.088814840" Nov 03 19:11:19.559051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:19.558961 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:20.041527 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.041493 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj"] Nov 03 19:11:20.045852 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.045832 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.104465 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.104432 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4320e5-openshift-default-dockercfg-mnmvp\"" Nov 03 19:11:20.206968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.206880 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:20.211970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.211461 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-certs\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.211970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.211532 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-envoy\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.211970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.211582 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-credential-socket\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.211970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.211625 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-data\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.212712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.212347 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-token\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.212712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.212423 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nbz\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-kube-api-access-h6nbz\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.212712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.212480 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istiod-ca-cert\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.212712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.212528 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-podinfo\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.212712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.212612 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-socket\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.284774 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.284735 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj"] Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316297 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-certs\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316371 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-envoy\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316424 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-credential-socket\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316474 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-data\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316528 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-token\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316566 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nbz\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-kube-api-access-h6nbz\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316619 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istiod-ca-cert\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316664 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-podinfo\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.316767 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-socket\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.317342 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-socket\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.317757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.317611 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-certs\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.318456 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.318217 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-credential-socket\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.318456 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.318282 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-data\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.320473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.318856 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istiod-ca-cert\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.320473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.320402 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-envoy\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.321171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.321146 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-podinfo\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.375542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.375512 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-token\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.375542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.375534 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nbz\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-kube-api-access-h6nbz\") pod \"e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:20.534334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.534278 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:20.662393 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:20.662304 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:21.094221 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:11:21.094190 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf929aba5_31a4_4bd0_a2bd_afc3de03159a.slice/crio-8a70f372a11ce29f5aa68841543d0b693145800252ac3a5d2584c263b53d1aed WatchSource:0}: Error finding container 8a70f372a11ce29f5aa68841543d0b693145800252ac3a5d2584c263b53d1aed: Status 404 returned error can't find the container with id 8a70f372a11ce29f5aa68841543d0b693145800252ac3a5d2584c263b53d1aed Nov 03 19:11:21.094221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:21.094196 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj"] Nov 03 19:11:21.097165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:21.097129 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:11:21.097275 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:21.097207 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:11:21.097339 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:21.097279 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:11:21.474853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:21.474670 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:22.010746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:22.010709 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" event={"ID":"f929aba5-31a4-4bd0-a2bd-afc3de03159a","Type":"ContainerStarted","Data":"cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8"} Nov 03 19:11:22.010927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:22.010763 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" event={"ID":"f929aba5-31a4-4bd0-a2bd-afc3de03159a","Type":"ContainerStarted","Data":"8a70f372a11ce29f5aa68841543d0b693145800252ac3a5d2584c263b53d1aed"} Nov 03 19:11:22.169918 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:22.169866 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" podStartSLOduration=3.169851234 podStartE2EDuration="3.169851234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:22.169435606 +0000 UTC m=+1957.050851709" watchObservedRunningTime="2026-11-03 19:11:22.169851234 +0000 UTC m=+1957.051267335" Nov 03 19:11:22.663472 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:22.663435 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:22.669470 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:22.669444 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:23.017541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:23.017472 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:23.018413 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:23.018392 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:11:24.362964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:24.362931 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p"] Nov 03 19:11:24.363409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:24.363154 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" containerID="cri-o://42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7" gracePeriod=30 Nov 03 19:11:25.050941 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:25.050910 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:25.344864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:25.344829 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:25.937790 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:11:25.937749 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:11:26.388767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:26.388733 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:26.957542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:26.957496 5988 patch_prober.go:29] interesting pod/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:11:26.957966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:26.957559 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:11:28.353349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:28.353313 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:28.556805 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:28.556758 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:29.629526 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.629465 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 19:11:29.650648 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.650624 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-envoy\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.650778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.650672 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb664c97-d248-4731-a27c-34b43cece904-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb664c97-d248-4731-a27c-34b43cece904-istiod-ca-cert\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.650778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.650732 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb664c97-d248-4731-a27c-34b43cece904-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb664c97-d248-4731-a27c-34b43cece904-istio-podinfo\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.650851 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.650804 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-data\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.650945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.650927 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-socket\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.651005 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.650965 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-kube-api-access-2mttx\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-kube-api-access-2mttx\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.651105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651089 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-certs\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.651162 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651129 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-credential-socket\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.651162 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651135 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-data" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:29.651261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651161 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-istio-token\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-istio-token\") pod \"bb664c97-d248-4731-a27c-34b43cece904\" (UID: \"bb664c97-d248-4731-a27c-34b43cece904\") " Nov 03 19:11:29.651261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651182 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-socket" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:29.651261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651188 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb664c97-d248-4731-a27c-34b43cece904-istiod-ca-cert" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:11:29.651425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651370 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-certs" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:29.651425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651392 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-credential-socket" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:29.651900 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651875 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.652021 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651905 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb664c97-d248-4731-a27c-34b43cece904-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.652021 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651920 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.652021 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651934 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.652021 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.651947 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.652926 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.652902 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-envoy" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:11:29.653037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.653009 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb664c97-d248-4731-a27c-34b43cece904-istio-podinfo" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:11:29.653148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.653124 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-kube-api-access-2mttx" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "kube-api-access-2mttx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:11:29.653305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.653289 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-istio-token" pod "bb664c97-d248-4731-a27c-34b43cece904" (UID: "bb664c97-d248-4731-a27c-34b43cece904"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:11:29.753802 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.753766 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb664c97-d248-4731-a27c-34b43cece904-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.753802 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.753799 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb664c97-d248-4731-a27c-34b43cece904-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.753802 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.753810 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mttx\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-kube-api-access-2mttx\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:29.754055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:29.753819 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb664c97-d248-4731-a27c-34b43cece904-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:11:30.002196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.002093 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:11:30.003596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.003573 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:11:30.072156 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.072119 5988 generic.go:356] "Generic (PLEG): container finished" podID="bb664c97-d248-4731-a27c-34b43cece904" containerID="42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7" exitCode=0 Nov 03 19:11:30.072376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.072151 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" event={"ID":"bb664c97-d248-4731-a27c-34b43cece904","Type":"ContainerDied","Data":"42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7"} Nov 03 19:11:30.072376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.072180 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" Nov 03 19:11:30.072376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.072214 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p" event={"ID":"bb664c97-d248-4731-a27c-34b43cece904","Type":"ContainerDied","Data":"c0c65b1e83d7ebb5e1a81727a05d1b0f530993ecbadc4d41252d9ef8e047bf78"} Nov 03 19:11:30.072376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.072238 5988 scope.go:122] "RemoveContainer" containerID="42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7" Nov 03 19:11:30.087420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.087401 5988 scope.go:122] "RemoveContainer" containerID="42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7" Nov 03 19:11:30.087812 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:11:30.087791 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7\": container with ID starting with 42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7 not found: ID does not exist" containerID="42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7" Nov 03 19:11:30.087894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.087826 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7"} err="failed to get container status \"42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7\": rpc error: code = NotFound desc = could not find container \"42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7\": container with ID starting with 42a51685a9d1ba3d52fcfdea5d64320bb4eb7ea9b616d82371ae4db820231ce7 not found: ID does not exist" Nov 03 19:11:30.327223 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.327189 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p"] Nov 03 19:11:30.373391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:30.373353 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-3a7056-openshift-default-85ccfddcf-2vv6p"] Nov 03 19:11:31.273811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:31.273777 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb664c97-d248-4731-a27c-34b43cece904" path="/var/lib/kubelet/pods/bb664c97-d248-4731-a27c-34b43cece904/volumes" Nov 03 19:11:31.615609 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:31.615573 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:36.013409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:36.013375 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:36.312787 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:36.312701 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:36.530494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:36.528852 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:41.048064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:41.048021 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:41.256199 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:41.256158 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:41.464089 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:41.464052 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:45.801941 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:45.801875 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:50.437762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:50.437724 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:52.156252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:52.156217 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:54.393266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:54.393192 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:54.591564 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:54.591502 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:55.030594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:55.030560 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:55.937055 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:11:55.937017 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:11:57.752358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:57.752317 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:11:59.627247 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:11:59.626843 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:01.101836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:01.101802 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:12:04.080764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.080732 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg"] Nov 03 19:12:04.081412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.081396 5988 cpu_manager.go:414] "containerMap: removing container" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" Nov 03 19:12:04.081466 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.081413 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" Nov 03 19:12:04.081566 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.081556 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb664c97-d248-4731-a27c-34b43cece904" containerName="istio-proxy" Nov 03 19:12:04.085953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.085932 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.137489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.137454 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-a93912-serving-cert\"" Nov 03 19:12:04.171738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.171702 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.171896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.171838 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.172006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.171986 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9dqx\" (UniqueName: \"kubernetes.io/projected/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-kube-api-access-c9dqx\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.172051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.172038 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.187449 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.187422 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg"] Nov 03 19:12:04.273341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.273304 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9dqx\" (UniqueName: \"kubernetes.io/projected/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-kube-api-access-c9dqx\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.273522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.273357 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.273522 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:12:04.273489 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-serving-cert: secret "maas-api-e2e-shared-b-w0-a93912-serving-cert" not found Nov 03 19:12:04.273646 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:12:04.273560 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls podName:1c89b584-f9f5-4ef6-8a1f-94586db8d09b nodeName:}" failed. No retries permitted until 2026-11-03 19:12:04.773544882 +0000 UTC m=+1999.654960962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls") pod "maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" (UID: "1c89b584-f9f5-4ef6-8a1f-94586db8d09b") : secret "maas-api-e2e-shared-b-w0-a93912-serving-cert" not found Nov 03 19:12:04.273646 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.273594 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.273793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.273670 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.274310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.274283 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.275891 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.275874 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.337245 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.337161 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:04.337414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.337384 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9dqx\" (UniqueName: \"kubernetes.io/projected/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-kube-api-access-c9dqx\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.553161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.553123 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:04.761373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.761299 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:04.782047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.782021 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.784246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.784220 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:04.966162 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.966132 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:04.999505 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:04.999475 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:05.372640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:05.372613 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg"] Nov 03 19:12:05.374617 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:12:05.374590 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c89b584_f9f5_4ef6_8a1f_94586db8d09b.slice/crio-f8fcc16a6198275480b7346d37b4a74cb5a05d56c369c235f90734953f45a0f2 WatchSource:0}: Error finding container f8fcc16a6198275480b7346d37b4a74cb5a05d56c369c235f90734953f45a0f2: Status 404 returned error can't find the container with id f8fcc16a6198275480b7346d37b4a74cb5a05d56c369c235f90734953f45a0f2 Nov 03 19:12:05.376397 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:05.376379 5988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:12:06.337973 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:06.337925 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" event={"ID":"1c89b584-f9f5-4ef6-8a1f-94586db8d09b","Type":"ContainerStarted","Data":"4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15"} Nov 03 19:12:06.338324 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:06.338283 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" event={"ID":"1c89b584-f9f5-4ef6-8a1f-94586db8d09b","Type":"ContainerStarted","Data":"f8fcc16a6198275480b7346d37b4a74cb5a05d56c369c235f90734953f45a0f2"} Nov 03 19:12:06.339613 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:06.339594 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:06.455677 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:06.455564 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" podStartSLOduration=1.660578927 podStartE2EDuration="2.455549379s" totalImagesPullingTime="794.970452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:06.452917461 +0000 UTC m=+2001.334333565" watchObservedRunningTime="2026-11-03 19:12:06.455549379 +0000 UTC m=+2001.336965481" Nov 03 19:12:08.177726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:08.177675 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:08.904621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:08.904588 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:09.877417 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:09.877378 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:10.203894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.203798 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx"] Nov 03 19:12:10.208924 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.208894 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.269648 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.258435 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-69e2b9-openshift-default-dockercfg-z4qvb\"" Nov 03 19:12:10.285668 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.285624 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-credential-socket\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.285890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.285672 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-certs\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.285890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.285772 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83803762-e846-4658-812c-a5dc412ebb1f-istio-podinfo\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.285890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.285879 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-istio-token\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.286047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.285920 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-socket\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.286101 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.286072 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-envoy\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.286200 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.286183 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-data\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.286260 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.286243 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83803762-e846-4658-812c-a5dc412ebb1f-istiod-ca-cert\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.286387 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.286368 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzrk\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-kube-api-access-7jzrk\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.309489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.309455 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx"] Nov 03 19:12:10.376346 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.376305 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:10.389549 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.389501 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzrk\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-kube-api-access-7jzrk\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.389853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.389636 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-credential-socket\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.390088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.389879 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-certs\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.390088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.389916 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83803762-e846-4658-812c-a5dc412ebb1f-istio-podinfo\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.390088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.390027 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-istio-token\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.390335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.390098 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-socket\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.390335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.390203 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-envoy\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.390335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.390279 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-data\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.390531 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.390334 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83803762-e846-4658-812c-a5dc412ebb1f-istiod-ca-cert\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.392328 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.392122 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-credential-socket\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.392453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.392365 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-socket\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.402438 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.401262 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-certs\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.408058 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.408027 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83803762-e846-4658-812c-a5dc412ebb1f-istiod-ca-cert\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.410895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.410617 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-data\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.410895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.410810 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-envoy\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.413143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.413117 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83803762-e846-4658-812c-a5dc412ebb1f-istio-podinfo\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.450120 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.450081 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzrk\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-kube-api-access-7jzrk\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.458187 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.458099 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-istio-token\") pod \"e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.524946 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.524904 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:10.603533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.603401 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:10.984482 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.984454 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx"] Nov 03 19:12:10.986756 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:12:10.986724 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83803762_e846_4658_812c_a5dc412ebb1f.slice/crio-6b0a33eb10c55e33782be1c31b343fa3711d72c566fabd8c817a56093d8f6aad WatchSource:0}: Error finding container 6b0a33eb10c55e33782be1c31b343fa3711d72c566fabd8c817a56093d8f6aad: Status 404 returned error can't find the container with id 6b0a33eb10c55e33782be1c31b343fa3711d72c566fabd8c817a56093d8f6aad Nov 03 19:12:10.989658 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.989622 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:12:10.989803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.989713 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:12:10.989803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:10.989747 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:12:11.422227 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:11.422186 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" event={"ID":"83803762-e846-4658-812c-a5dc412ebb1f","Type":"ContainerStarted","Data":"0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f"} Nov 03 19:12:11.422427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:11.422247 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" event={"ID":"83803762-e846-4658-812c-a5dc412ebb1f","Type":"ContainerStarted","Data":"6b0a33eb10c55e33782be1c31b343fa3711d72c566fabd8c817a56093d8f6aad"} Nov 03 19:12:11.525584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:11.525554 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:11.537608 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:11.537557 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podStartSLOduration=1.537542423 podStartE2EDuration="1.537542423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:11.536056003 +0000 UTC m=+2006.417472126" watchObservedRunningTime="2026-11-03 19:12:11.537542423 +0000 UTC m=+2006.418958525" Nov 03 19:12:11.954107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:11.954065 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62"] Nov 03 19:12:11.958834 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:11.958805 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.008625 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.008584 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-97e020-openshift-default-dockercfg-htkrg\"" Nov 03 19:12:12.059245 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.059207 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62"] Nov 03 19:12:12.115380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115337 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.115608 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115414 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.115608 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115521 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.115837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115617 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.115837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115659 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.115837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115794 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.115993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115845 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de86ec7-7982-4771-a8a8-c62921c86eb6-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.116061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.115996 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.116061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.116045 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdtl\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-kube-api-access-zsdtl\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217270 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217310 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217340 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217530 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217743 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217719 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217779 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de86ec7-7982-4771-a8a8-c62921c86eb6-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217871 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217891 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.217962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.217949 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdtl\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-kube-api-access-zsdtl\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.218071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.218045 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.218141 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.218121 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-credential-socket\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.218195 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.218155 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-data\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.218398 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.218378 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-certs\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.218619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.218597 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de86ec7-7982-4771-a8a8-c62921c86eb6-istiod-ca-cert\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.220172 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.220147 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-envoy\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.220532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.220514 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-podinfo\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.275837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.275802 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-token\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.275992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.275956 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdtl\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-kube-api-access-zsdtl\") pod \"e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:12.531904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.531831 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:12.575386 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:12.575345 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:13.002869 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.002838 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62"] Nov 03 19:12:13.004588 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:12:13.004557 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de86ec7_7982_4771_a8a8_c62921c86eb6.slice/crio-8bb2d647dd14a282e827f1b572a0c1020c0577832b8c8c1332eca2ad2039d7c0 WatchSource:0}: Error finding container 8bb2d647dd14a282e827f1b572a0c1020c0577832b8c8c1332eca2ad2039d7c0: Status 404 returned error can't find the container with id 8bb2d647dd14a282e827f1b572a0c1020c0577832b8c8c1332eca2ad2039d7c0 Nov 03 19:12:13.007795 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.007762 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:12:13.007879 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.007844 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:12:13.007916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.007876 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:12:13.356722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.356694 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:12:13.442838 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.442798 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" event={"ID":"5de86ec7-7982-4771-a8a8-c62921c86eb6","Type":"ContainerStarted","Data":"a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5"} Nov 03 19:12:13.443025 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.442859 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" event={"ID":"5de86ec7-7982-4771-a8a8-c62921c86eb6","Type":"ContainerStarted","Data":"8bb2d647dd14a282e827f1b572a0c1020c0577832b8c8c1332eca2ad2039d7c0"} Nov 03 19:12:13.443321 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.443290 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:13.446226 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.446199 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:12:13.575824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.575786 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:13.717357 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:13.717237 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podStartSLOduration=2.7172217549999997 podStartE2EDuration="2.717221755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:13.715860865 +0000 UTC m=+2008.597276967" watchObservedRunningTime="2026-11-03 19:12:13.717221755 +0000 UTC m=+2008.598637856" Nov 03 19:12:14.372305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:14.372225 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:14.578268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:14.578228 5988 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:12:14.578433 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:14.578287 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:12:14.616504 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:14.616452 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:14.894709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:14.880929 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:15.576297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:15.576256 5988 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:12:15.576747 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:15.576314 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:12:16.576248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:16.576212 5988 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:12:16.576439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:16.576283 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:12:17.576227 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:17.576170 5988 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:12:17.576461 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:17.576245 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:12:18.576334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:18.576293 5988 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:12:18.576508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:18.576383 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:12:19.576287 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:19.576247 5988 patch_prober.go:29] interesting pod/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 19:12:19.576480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:19.576310 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 19:12:19.912198 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:19.912091 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:20.580621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:20.580583 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:20.581011 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:20.580987 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:20.581840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:20.581819 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:12:20.896978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:20.896892 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:21.104593 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:21.104548 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:24.829716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:24.829314 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:25.938649 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:12:25.938608 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:12:29.318554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:29.318508 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:29.528086 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:29.528050 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:29.737844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:29.737752 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:34.013097 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:34.013049 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:38.622234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:38.622172 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:43.314798 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:43.314760 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:47.759706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:47.752435 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:48.025859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:48.025772 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:50.780477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:50.780439 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:12:52.712917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:52.712875 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:53.068365 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:53.066483 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:55.938106 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:12:55.938059 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:12:57.822968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:57.822935 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:12:59.579218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:12:59.579157 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:02.292162 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:02.292102 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:02.498847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:02.498811 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:07.218444 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:07.218382 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:11.916639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:11.916595 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:15.075044 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.075004 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj"] Nov 03 19:13:15.080183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.080162 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.127189 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.127155 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj"] Nov 03 19:13:15.169760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.169732 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:13:15.169911 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.169796 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:13:15.241225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.241197 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkqx\" (UniqueName: \"kubernetes.io/projected/fbfe2657-334b-4267-aa77-4a284674c162-kube-api-access-fpkqx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.241384 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.241267 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.241497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.241481 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.342610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.342543 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.342610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.342583 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkqx\" (UniqueName: \"kubernetes.io/projected/fbfe2657-334b-4267-aa77-4a284674c162-kube-api-access-fpkqx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.342807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.342644 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.343345 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.343317 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.344942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.344922 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.399936 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.399907 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkqx\" (UniqueName: \"kubernetes.io/projected/fbfe2657-334b-4267-aa77-4a284674c162-kube-api-access-fpkqx\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.676969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.676890 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:15.695380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.695355 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:15.882064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:15.882019 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:16.065929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:16.065903 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj"] Nov 03 19:13:16.067819 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:13:16.067786 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfe2657_334b_4267_aa77_4a284674c162.slice/crio-a1b096a48a3e7d746ac382b12ef73d5b9b3f975b58b64f0461f2378158c2abc4 WatchSource:0}: Error finding container a1b096a48a3e7d746ac382b12ef73d5b9b3f975b58b64f0461f2378158c2abc4: Status 404 returned error can't find the container with id a1b096a48a3e7d746ac382b12ef73d5b9b3f975b58b64f0461f2378158c2abc4 Nov 03 19:13:16.546822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:16.546781 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:16.892289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:16.892245 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" event={"ID":"fbfe2657-334b-4267-aa77-4a284674c162","Type":"ContainerStarted","Data":"37ff6cf674e003333da3a661ec0bf3789b972b7ee3f010477e7eaaa9f8eb6a6c"} Nov 03 19:13:16.892461 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:16.892311 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" event={"ID":"fbfe2657-334b-4267-aa77-4a284674c162","Type":"ContainerStarted","Data":"a1b096a48a3e7d746ac382b12ef73d5b9b3f975b58b64f0461f2378158c2abc4"} Nov 03 19:13:17.003953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:17.003899 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" podStartSLOduration=2.00388488 podStartE2EDuration="2.00388488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:17.000844817 +0000 UTC m=+2071.882260919" watchObservedRunningTime="2026-11-03 19:13:17.00388488 +0000 UTC m=+2071.885300982" Nov 03 19:13:17.900090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:17.900053 5988 generic.go:356] "Generic (PLEG): container finished" podID="fbfe2657-334b-4267-aa77-4a284674c162" containerID="37ff6cf674e003333da3a661ec0bf3789b972b7ee3f010477e7eaaa9f8eb6a6c" exitCode=0 Nov 03 19:13:17.900500 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:17.900135 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" event={"ID":"fbfe2657-334b-4267-aa77-4a284674c162","Type":"ContainerDied","Data":"37ff6cf674e003333da3a661ec0bf3789b972b7ee3f010477e7eaaa9f8eb6a6c"} Nov 03 19:13:19.067576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.067544 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:19.197987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.197909 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfe2657-334b-4267-aa77-4a284674c162-kube-api-access-fpkqx\" (UniqueName: \"kubernetes.io/projected/fbfe2657-334b-4267-aa77-4a284674c162-kube-api-access-fpkqx\") pod \"fbfe2657-334b-4267-aa77-4a284674c162\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " Nov 03 19:13:19.198152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.198059 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-openshift-ca\") pod \"fbfe2657-334b-4267-aa77-4a284674c162\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " Nov 03 19:13:19.198152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.198105 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-xks-ca\") pod \"fbfe2657-334b-4267-aa77-4a284674c162\" (UID: \"fbfe2657-334b-4267-aa77-4a284674c162\") " Nov 03 19:13:19.198490 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.198454 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-openshift-ca" pod "fbfe2657-334b-4267-aa77-4a284674c162" (UID: "fbfe2657-334b-4267-aa77-4a284674c162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:19.198728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.198701 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:19.200814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.200787 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfe2657-334b-4267-aa77-4a284674c162-kube-api-access-fpkqx" pod "fbfe2657-334b-4267-aa77-4a284674c162" (UID: "fbfe2657-334b-4267-aa77-4a284674c162"). InnerVolumeSpecName "kube-api-access-fpkqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:19.201258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.201230 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-xks-ca" pod "fbfe2657-334b-4267-aa77-4a284674c162" (UID: "fbfe2657-334b-4267-aa77-4a284674c162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:13:19.300919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.300894 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfe2657-334b-4267-aa77-4a284674c162-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:19.301124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.301097 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkqx\" (UniqueName: \"kubernetes.io/projected/fbfe2657-334b-4267-aa77-4a284674c162-kube-api-access-fpkqx\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:19.914545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.914514 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" Nov 03 19:13:19.914738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.914542 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj" event={"ID":"fbfe2657-334b-4267-aa77-4a284674c162","Type":"ContainerDied","Data":"a1b096a48a3e7d746ac382b12ef73d5b9b3f975b58b64f0461f2378158c2abc4"} Nov 03 19:13:19.914738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:19.914592 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b096a48a3e7d746ac382b12ef73d5b9b3f975b58b64f0461f2378158c2abc4" Nov 03 19:13:21.344253 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:21.344197 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:25.937499 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:25.937419 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:13:26.321794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:26.318215 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:28.243333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:28.243302 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:28.534299 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:28.534204 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:28.746570 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:28.746531 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:28.960706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:28.958456 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:29.217428 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:29.217334 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:31.391669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:31.391632 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:32.645391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:32.645362 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:32.864376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:32.864321 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:35.899954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:35.899916 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:36.116824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.116789 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:36.539705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.536892 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:36.879673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.879643 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd"] Nov 03 19:13:36.880449 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.880431 5988 cpu_manager.go:414] "containerMap: removing container" podUID="fbfe2657-334b-4267-aa77-4a284674c162" containerName="revoke-keys" Nov 03 19:13:36.880498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.880452 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfe2657-334b-4267-aa77-4a284674c162" containerName="revoke-keys" Nov 03 19:13:36.880566 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.880556 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfe2657-334b-4267-aa77-4a284674c162" containerName="revoke-keys" Nov 03 19:13:36.884151 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.884130 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:36.916302 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.916279 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:36.916663 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.916452 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5pm\" (UniqueName: \"kubernetes.io/projected/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-kube-api-access-2q5pm\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:36.916663 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.916510 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:36.980768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.980734 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:13:36.980768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.980738 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:13:36.983975 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:36.983952 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd"] Nov 03 19:13:37.017362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.017332 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:37.017522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.017503 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5pm\" (UniqueName: \"kubernetes.io/projected/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-kube-api-access-2q5pm\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:37.017628 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.017611 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:37.018149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.018127 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:37.019923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.019903 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:37.074916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.074889 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5pm\" (UniqueName: \"kubernetes.io/projected/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-kube-api-access-2q5pm\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:37.197535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.197459 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:37.568999 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:37.568967 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd"] Nov 03 19:13:37.570157 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:13:37.570119 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae618a5_5a31_4cd9_99c1_d77d2899e8ab.slice/crio-346da97661697d352eca83e873fc690a28afd26c566b44ae7c7c2dd9aa3899a7 WatchSource:0}: Error finding container 346da97661697d352eca83e873fc690a28afd26c566b44ae7c7c2dd9aa3899a7: Status 404 returned error can't find the container with id 346da97661697d352eca83e873fc690a28afd26c566b44ae7c7c2dd9aa3899a7 Nov 03 19:13:38.105579 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:38.105468 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" event={"ID":"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab","Type":"ContainerStarted","Data":"1550da91a84b9dfabdcdfd4a007f068ea25ce4e962a9a368bc7179aa3020a197"} Nov 03 19:13:38.105579 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:38.105567 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" event={"ID":"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab","Type":"ContainerStarted","Data":"346da97661697d352eca83e873fc690a28afd26c566b44ae7c7c2dd9aa3899a7"} Nov 03 19:13:38.216713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:38.216648 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" podStartSLOduration=2.216634244 podStartE2EDuration="2.216634244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:38.21362021 +0000 UTC m=+2093.095036313" watchObservedRunningTime="2026-11-03 19:13:38.216634244 +0000 UTC m=+2093.098050388" Nov 03 19:13:39.114228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.114190 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:13:39.116110 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.116076 5988 generic.go:356] "Generic (PLEG): container finished" podID="8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" containerID="1550da91a84b9dfabdcdfd4a007f068ea25ce4e962a9a368bc7179aa3020a197" exitCode=0 Nov 03 19:13:39.116269 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.116134 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" event={"ID":"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab","Type":"ContainerDied","Data":"1550da91a84b9dfabdcdfd4a007f068ea25ce4e962a9a368bc7179aa3020a197"} Nov 03 19:13:39.281065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.281037 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr"] Nov 03 19:13:39.281497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.281462 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" containerID="cri-o://dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1" gracePeriod=30 Nov 03 19:13:39.285061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.285039 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn"] Nov 03 19:13:39.285366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.285240 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" containerID="cri-o://e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0" gracePeriod=30 Nov 03 19:13:39.465079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.464092 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq"] Nov 03 19:13:39.465079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.464328 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" containerID="cri-o://0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f" gracePeriod=30 Nov 03 19:13:39.473097 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.472345 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd"] Nov 03 19:13:39.473097 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:39.472601 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" containerID="cri-o://3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441" gracePeriod=30 Nov 03 19:13:40.367613 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.367590 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:40.501561 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.501470 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-xks-ca\") pod \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " Nov 03 19:13:40.501561 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.501540 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-openshift-ca\") pod \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " Nov 03 19:13:40.501824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.501641 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-kube-api-access-2q5pm\" (UniqueName: \"kubernetes.io/projected/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-kube-api-access-2q5pm\") pod \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\" (UID: \"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab\") " Nov 03 19:13:40.502046 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.502015 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-openshift-ca" pod "8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" (UID: "8ae618a5-5a31-4cd9-99c1-d77d2899e8ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:40.502424 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.502397 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:40.503868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.503834 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-xks-ca" pod "8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" (UID: "8ae618a5-5a31-4cd9-99c1-d77d2899e8ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:13:40.503979 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.503939 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-kube-api-access-2q5pm" pod "8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" (UID: "8ae618a5-5a31-4cd9-99c1-d77d2899e8ab"). InnerVolumeSpecName "kube-api-access-2q5pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:40.604391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.604355 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:40.604391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.604388 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5pm\" (UniqueName: \"kubernetes.io/projected/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab-kube-api-access-2q5pm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:40.771982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.771888 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:40.984733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:40.984702 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:41.179047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:41.179017 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" Nov 03 19:13:41.179221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:41.179059 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd" event={"ID":"8ae618a5-5a31-4cd9-99c1-d77d2899e8ab","Type":"ContainerDied","Data":"346da97661697d352eca83e873fc690a28afd26c566b44ae7c7c2dd9aa3899a7"} Nov 03 19:13:41.179221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:41.179111 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346da97661697d352eca83e873fc690a28afd26c566b44ae7c7c2dd9aa3899a7" Nov 03 19:13:41.620181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:41.616835 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:42.445658 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:42.445619 5988 patch_prober.go:29] interesting pod/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:13:42.445894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:42.445717 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:13:43.844395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:43.844362 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b"] Nov 03 19:13:43.844794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:43.844576 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" podUID="4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" containerName="discovery" containerID="cri-o://7e00eb8b6d8861ea8211d2ec17061ba3d876149ccf0ca2abb47e89d596c2d84c" gracePeriod=30 Nov 03 19:13:44.210894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.210598 5988 generic.go:356] "Generic (PLEG): container finished" podID="4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" containerID="7e00eb8b6d8861ea8211d2ec17061ba3d876149ccf0ca2abb47e89d596c2d84c" exitCode=0 Nov 03 19:13:44.210894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.210839 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" event={"ID":"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad","Type":"ContainerDied","Data":"7e00eb8b6d8861ea8211d2ec17061ba3d876149ccf0ca2abb47e89d596c2d84c"} Nov 03 19:13:44.278001 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.277971 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:13:44.395160 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.395127 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-dns-cert\") pod \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " Nov 03 19:13:44.395310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.395173 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-local-certs\") pod \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " Nov 03 19:13:44.395460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.395380 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-ca-configmap\") pod \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " Nov 03 19:13:44.395460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.395425 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-kubeconfig\") pod \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " Nov 03 19:13:44.395592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.395494 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-token\") pod \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " Nov 03 19:13:44.395592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.395556 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-kube-api-access-hc2kg\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-kube-api-access-hc2kg\") pod \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " Nov 03 19:13:44.396310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.395762 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-cacerts\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-cacerts\") pod \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\" (UID: \"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad\") " Nov 03 19:13:44.397305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.397070 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-ca-configmap" pod "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" (UID: "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:44.404063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.404039 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-kubeconfig" pod "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" (UID: "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:13:44.404152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.404042 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-kube-api-access-hc2kg" pod "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" (UID: "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad"). InnerVolumeSpecName "kube-api-access-hc2kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:44.404152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.404058 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-dns-cert" pod "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" (UID: "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:13:44.404152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.404065 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-local-certs" pod "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" (UID: "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.404152 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.404061 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-token" pod "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" (UID: "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:44.406202 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.406074 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-cacerts" pod "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" (UID: "4ad41e61-e71e-4735-9c7d-a07fbf7df6ad"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:13:44.498303 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.498274 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.498420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.498307 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2kg\" (UniqueName: \"kubernetes.io/projected/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-kube-api-access-hc2kg\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.498420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.498325 5988 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-cacerts\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.498420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.498338 5988 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-dns-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.498420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.498351 5988 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-local-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.498420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.498366 5988 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-csr-ca-configmap\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.498420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.498381 5988 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad-istio-kubeconfig\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.616660 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.616626 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:13:44.803373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803267 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-credential-socket\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803332 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-podinfo\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803408 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-token\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-token\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803488 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istiod-ca-cert\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803539 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-kube-api-access-wz4tq\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-kube-api-access-wz4tq\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803758 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803592 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-data\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803758 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803652 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-envoy\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803760 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-socket\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.803931 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.803864 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-certs\") pod \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\" (UID: \"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d\") " Nov 03 19:13:44.804539 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.804484 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istiod-ca-cert" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:44.806086 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.805811 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-socket" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.806086 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.805827 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-data" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.807621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.807015 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-credential-socket" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.808466 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.808414 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-certs" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.811935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.811582 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 19:13:44.814717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.814546 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-envoy" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.814717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.814646 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-podinfo" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:44.816487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.816461 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-token" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:44.817884 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.817823 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-kube-api-access-wz4tq" pod "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" (UID: "78b3c84c-27c0-4de1-8f48-8b8a07c63c3d"). InnerVolumeSpecName "kube-api-access-wz4tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:44.889775 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.887999 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:13:44.905697 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.905664 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-socket\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.905794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.905770 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-kube-api-access-lw68d\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-kube-api-access-lw68d\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.905838 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.905810 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-certs\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.905871 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.905864 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-token\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-token\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.905982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.905965 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-credential-socket\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.906046 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906035 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-envoy\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.906084 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906060 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-podinfo\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.906127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906085 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istiod-ca-cert\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.906158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906125 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-data\") pod \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\" (UID: \"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420\") " Nov 03 19:13:44.906829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906801 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.906928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906836 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4tq\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-kube-api-access-wz4tq\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.906928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906854 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.906928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906869 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.906928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906883 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.906928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906903 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.906928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906924 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.907304 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906938 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.907304 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.906954 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:44.907304 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.907143 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-socket" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.907576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.907539 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istiod-ca-cert" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:44.907839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.907813 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-certs" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.907927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.907883 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-credential-socket" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.908010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.907990 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-data" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.908584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.908559 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-token" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:44.908964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.908936 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-kube-api-access-lw68d" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "kube-api-access-lw68d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:44.909459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.909438 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-podinfo" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:44.910134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.910116 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-envoy" pod "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" (UID: "c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:44.994234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:44.994205 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 19:13:45.008050 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008022 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-token\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-token\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008165 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-podinfo\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008254 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-kube-api-access-xfshd\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-kube-api-access-xfshd\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008321 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008281 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-certs\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008355 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008337 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-envoy\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008389 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d618dc2-5662-46bb-b1b6-9824a6e01924-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d618dc2-5662-46bb-b1b6-9824a6e01924-istiod-ca-cert\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008417 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-credential-socket\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008633 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008526 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-socket\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008633 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008581 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-data\") pod \"3d618dc2-5662-46bb-b1b6-9824a6e01924\" (UID: \"3d618dc2-5662-46bb-b1b6-9824a6e01924\") " Nov 03 19:13:45.008883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.008836 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-certs" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.009031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009010 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-data" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.009132 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009104 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d618dc2-5662-46bb-b1b6-9824a6e01924-istiod-ca-cert" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:45.009436 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009332 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-credential-socket" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.009541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009454 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-socket" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009600 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009623 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009639 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009653 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009662 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d618dc2-5662-46bb-b1b6-9824a6e01924-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009670 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009705 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw68d\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-kube-api-access-lw68d\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009718 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009730 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009743 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009755 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009768 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009780 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.009813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.009793 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.010786 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.010747 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-token" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:45.011639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.011616 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-kube-api-access-xfshd" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "kube-api-access-xfshd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:45.011878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.011848 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-podinfo" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:45.012016 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.011991 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-envoy" pod "3d618dc2-5662-46bb-b1b6-9824a6e01924" (UID: "3d618dc2-5662-46bb-b1b6-9824a6e01924"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.110891 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-podinfo\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.110960 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-socket\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.111017 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-envoy\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.111049 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-certs\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.111103 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-kube-api-access-cc8pw\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-kube-api-access-cc8pw\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.111144 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istiod-ca-cert\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.111321 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-credential-socket\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.111352 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-data\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.111480 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-token\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-token\") pod \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\" (UID: \"f4adea64-c229-4ae9-9ee1-bb740a9cad41\") " Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.112220 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.112246 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.112262 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfshd\" (UniqueName: \"kubernetes.io/projected/3d618dc2-5662-46bb-b1b6-9824a6e01924-kube-api-access-xfshd\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.112744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.112277 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d618dc2-5662-46bb-b1b6-9824a6e01924-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.113832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.113572 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-certs" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.113832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.113661 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-socket" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.114033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.113926 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-credential-socket" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.114033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.114007 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-data" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.114145 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.114097 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istiod-ca-cert" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:13:45.115104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.115070 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-token" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:45.116244 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.116210 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-envoy" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:13:45.117068 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.117041 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-kube-api-access-cc8pw" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "kube-api-access-cc8pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:13:45.117278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.117244 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-podinfo" pod "f4adea64-c229-4ae9-9ee1-bb740a9cad41" (UID: "f4adea64-c229-4ae9-9ee1-bb740a9cad41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213637 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213668 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213677 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213701 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213710 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213724 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213736 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4adea64-c229-4ae9-9ee1-bb740a9cad41-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213744 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8pw\" (UniqueName: \"kubernetes.io/projected/f4adea64-c229-4ae9-9ee1-bb740a9cad41-kube-api-access-cc8pw\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.213749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.213752 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4adea64-c229-4ae9-9ee1-bb740a9cad41-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:13:45.219260 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.219212 5988 generic.go:356] "Generic (PLEG): container finished" podID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerID="3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441" exitCode=0 Nov 03 19:13:45.219639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.219586 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" event={"ID":"f4adea64-c229-4ae9-9ee1-bb740a9cad41","Type":"ContainerDied","Data":"3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441"} Nov 03 19:13:45.219878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.219828 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" event={"ID":"f4adea64-c229-4ae9-9ee1-bb740a9cad41","Type":"ContainerDied","Data":"2d8b8480a7af13235f8e1d4ffe3609802905eb3150e05c18529b80ce61fcace5"} Nov 03 19:13:45.220035 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.219972 5988 scope.go:122] "RemoveContainer" containerID="3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441" Nov 03 19:13:45.220347 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.220277 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd" Nov 03 19:13:45.226196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.226104 5988 generic.go:356] "Generic (PLEG): container finished" podID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerID="dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1" exitCode=0 Nov 03 19:13:45.226196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.226187 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" event={"ID":"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d","Type":"ContainerDied","Data":"dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1"} Nov 03 19:13:45.226354 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.226232 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" event={"ID":"78b3c84c-27c0-4de1-8f48-8b8a07c63c3d","Type":"ContainerDied","Data":"20a29693ce1ec27f135b581a8546e5aa88148b906b493f43b4d5182c95c5fc10"} Nov 03 19:13:45.226786 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.226659 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr" Nov 03 19:13:45.228000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.227979 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" Nov 03 19:13:45.228182 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.228112 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b" event={"ID":"4ad41e61-e71e-4735-9c7d-a07fbf7df6ad","Type":"ContainerDied","Data":"e719dd464e4642fd3a30530360d59f98ee2544964a674524f76b915c28983ad0"} Nov 03 19:13:45.232725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.230986 5988 generic.go:356] "Generic (PLEG): container finished" podID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerID="0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f" exitCode=0 Nov 03 19:13:45.232725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.231076 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" event={"ID":"3d618dc2-5662-46bb-b1b6-9824a6e01924","Type":"ContainerDied","Data":"0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f"} Nov 03 19:13:45.232725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.231118 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" event={"ID":"3d618dc2-5662-46bb-b1b6-9824a6e01924","Type":"ContainerDied","Data":"7da98153ed2076dc1db8aed269ea6b4ce0ae7b69d1a95ad1158feebbbc60a705"} Nov 03 19:13:45.232725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.231194 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq" Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.239809 5988 generic.go:356] "Generic (PLEG): container finished" podID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerID="e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0" exitCode=0 Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.239968 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" event={"ID":"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420","Type":"ContainerDied","Data":"e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0"} Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.240004 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" event={"ID":"c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420","Type":"ContainerDied","Data":"e242203a2dfc0381c6bfb50e3abd81e18ff243432f1185d58210a322c8dcd5ec"} Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.240075 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn" Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.241371 5988 scope.go:122] "RemoveContainer" containerID="3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441" Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:45.241669 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441\": container with ID starting with 3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441 not found: ID does not exist" containerID="3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441" Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.241720 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441"} err="failed to get container status \"3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441\": rpc error: code = NotFound desc = could not find container \"3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441\": container with ID starting with 3a685c29b16018c27fa2b7b336e03f00b5a0e50d72697265766a0ae66ab7e441 not found: ID does not exist" Nov 03 19:13:45.244693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.241742 5988 scope.go:122] "RemoveContainer" containerID="dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1" Nov 03 19:13:45.286760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.284806 5988 scope.go:122] "RemoveContainer" containerID="dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1" Nov 03 19:13:45.286760 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:45.285147 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1\": container with ID starting with dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1 not found: ID does not exist" containerID="dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1" Nov 03 19:13:45.286760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.285179 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1"} err="failed to get container status \"dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1\": rpc error: code = NotFound desc = could not find container \"dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1\": container with ID starting with dff74ffc2399fd6e6f436e24c5e2991f6a3f2d81cdd95704cc4c4a290c1183c1 not found: ID does not exist" Nov 03 19:13:45.286760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.285201 5988 scope.go:122] "RemoveContainer" containerID="7e00eb8b6d8861ea8211d2ec17061ba3d876149ccf0ca2abb47e89d596c2d84c" Nov 03 19:13:45.301531 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.301503 5988 scope.go:122] "RemoveContainer" containerID="0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f" Nov 03 19:13:45.310836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.310817 5988 scope.go:122] "RemoveContainer" containerID="0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f" Nov 03 19:13:45.311131 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:45.311107 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f\": container with ID starting with 0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f not found: ID does not exist" containerID="0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f" Nov 03 19:13:45.311171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.311139 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f"} err="failed to get container status \"0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f\": rpc error: code = NotFound desc = could not find container \"0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f\": container with ID starting with 0afc39baeeae28019a70ffc315cd0ee84fb2bb892a0a3468fcdea1ee3edf345f not found: ID does not exist" Nov 03 19:13:45.311171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.311156 5988 scope.go:122] "RemoveContainer" containerID="e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0" Nov 03 19:13:45.320033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.320011 5988 scope.go:122] "RemoveContainer" containerID="e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0" Nov 03 19:13:45.320298 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:45.320278 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0\": container with ID starting with e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0 not found: ID does not exist" containerID="e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0" Nov 03 19:13:45.320358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.320305 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0"} err="failed to get container status \"e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0\": rpc error: code = NotFound desc = could not find container \"e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0\": container with ID starting with e302afdd27d52f77e8b27f4b5dc634b759da69d688cda0e113b4a7d08053c8b0 not found: ID does not exist" Nov 03 19:13:45.485194 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.485093 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd"] Nov 03 19:13:45.532113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.532082 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-ntspd"] Nov 03 19:13:45.697078 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.697048 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b"] Nov 03 19:13:45.721639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.721612 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xl65b"] Nov 03 19:13:45.831994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.831966 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr"] Nov 03 19:13:45.836800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.836778 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-5ktvr"] Nov 03 19:13:45.940159 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.940128 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq"] Nov 03 19:13:45.974472 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:45.974440 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-d5ppq"] Nov 03 19:13:46.060054 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:46.059967 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn"] Nov 03 19:13:46.062549 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:46.062525 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-d7xjn"] Nov 03 19:13:46.574604 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:46.574572 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:47.279756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:47.279714 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" path="/var/lib/kubelet/pods/3d618dc2-5662-46bb-b1b6-9824a6e01924/volumes" Nov 03 19:13:47.280403 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:47.280375 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" path="/var/lib/kubelet/pods/4ad41e61-e71e-4735-9c7d-a07fbf7df6ad/volumes" Nov 03 19:13:47.281090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:47.281069 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" path="/var/lib/kubelet/pods/78b3c84c-27c0-4de1-8f48-8b8a07c63c3d/volumes" Nov 03 19:13:47.281870 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:47.281855 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" path="/var/lib/kubelet/pods/c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420/volumes" Nov 03 19:13:47.282300 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:47.282286 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" path="/var/lib/kubelet/pods/f4adea64-c229-4ae9-9ee1-bb740a9cad41/volumes" Nov 03 19:13:48.884642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:48.884605 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:49.088494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:49.088461 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:50.278376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:50.278341 5988 generic.go:356] "Generic (PLEG): container finished" podID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerID="cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8" exitCode=0 Nov 03 19:13:50.278889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:50.278400 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" event={"ID":"f929aba5-31a4-4bd0-a2bd-afc3de03159a","Type":"ContainerDied","Data":"cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8"} Nov 03 19:13:50.278970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:50.278941 5988 scope.go:122] "RemoveContainer" containerID="cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8" Nov 03 19:13:50.281199 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:50.281155 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:13:50.281335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:50.281230 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:13:50.281335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:50.281267 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:13:51.288552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:51.288519 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" event={"ID":"f929aba5-31a4-4bd0-a2bd-afc3de03159a","Type":"ContainerStarted","Data":"6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e"} Nov 03 19:13:51.664029 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:51.663995 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:13:51.664385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:51.664361 5988 patch_prober.go:29] interesting pod/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:13:51.664555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:51.664531 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 19:13:51.982502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:51.982372 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:52.665117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:52.665075 5988 patch_prober.go:29] interesting pod/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:13:52.665783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:52.665748 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 19:13:53.018161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:53.018079 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:13:53.082657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:53.082605 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:53.671712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:53.671656 5988 patch_prober.go:29] interesting pod/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:13:53.672202 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:53.671739 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 19:13:54.671641 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:54.671600 5988 patch_prober.go:29] interesting pod/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:13:54.671849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:54.671672 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 19:13:55.670830 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:55.670791 5988 patch_prober.go:29] interesting pod/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 19:13:55.671047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:55.670844 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 19:13:55.936510 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:55.936421 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:13:56.666811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.666785 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:13:56.668008 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.667986 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:13:56.707860 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.707825 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr"] Nov 03 19:13:56.708574 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708553 5988 cpu_manager.go:414] "containerMap: removing container" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708575 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708592 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708597 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708613 5988 cpu_manager.go:414] "containerMap: removing container" podUID="8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" containerName="revoke-keys" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708618 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" containerName="revoke-keys" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708638 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708643 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708649 5988 cpu_manager.go:414] "containerMap: removing container" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" Nov 03 19:13:56.708653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708653 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708670 5988 cpu_manager.go:414] "containerMap: removing container" podUID="4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" containerName="discovery" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708674 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" containerName="discovery" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708828 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d618dc2-5662-46bb-b1b6-9824a6e01924" containerName="istio-proxy" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708840 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b3c84c-27c0-4de1-8f48-8b8a07c63c3d" containerName="istio-proxy" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708851 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad41e61-e71e-4735-9c7d-a07fbf7df6ad" containerName="discovery" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708874 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ec5ec5-d0de-4dd6-8f2d-164bdef7c420" containerName="istio-proxy" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708882 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4adea64-c229-4ae9-9ee1-bb740a9cad41" containerName="istio-proxy" Nov 03 19:13:56.709105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.708891 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" containerName="revoke-keys" Nov 03 19:13:56.711842 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.711822 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.760960 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.760936 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-69e2b9-serving-cert\"" Nov 03 19:13:56.770762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.770736 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr"] Nov 03 19:13:56.818406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.818372 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-metrics-certs\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.818619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.818575 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s29z2\" (UniqueName: \"kubernetes.io/projected/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-kube-api-access-s29z2\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.818942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.818919 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.819019 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.819004 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-openshift-service-ca\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.919993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.919909 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.920136 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.919995 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-openshift-service-ca\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.920136 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:56.920059 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-serving-cert: secret "maas-api-e2e-api-b-69e2b9-serving-cert" not found Nov 03 19:13:56.920136 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.920104 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-metrics-certs\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.920262 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:13:56.920141 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls podName:a6e29b9f-c2a3-464e-9e89-22e3aefad77c nodeName:}" failed. No retries permitted until 2026-11-03 19:13:57.420121611 +0000 UTC m=+2112.301537704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls") pod "maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" (UID: "a6e29b9f-c2a3-464e-9e89-22e3aefad77c") : secret "maas-api-e2e-api-b-69e2b9-serving-cert" not found Nov 03 19:13:56.920262 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.920231 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s29z2\" (UniqueName: \"kubernetes.io/projected/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-kube-api-access-s29z2\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.920642 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.920621 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-openshift-service-ca\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.922448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.922426 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-metrics-certs\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:56.976032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.976000 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:56.978010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:56.977988 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s29z2\" (UniqueName: \"kubernetes.io/projected/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-kube-api-access-s29z2\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:57.184583 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.184474 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:57.410784 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.410746 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:13:57.427390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.427346 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:57.429753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.429727 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls\") pod \"maas-api-e2e-api-b-69e2b9-5657568589-r8nxr\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:57.489800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.489731 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj"] Nov 03 19:13:57.493343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.493325 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.544885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.544859 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:13:57.545305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.545277 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:13:57.548528 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.548506 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj"] Nov 03 19:13:57.625745 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.625706 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:57.629636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.629606 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g85\" (UniqueName: \"kubernetes.io/projected/68654e8e-63aa-4547-bd5c-11e0be2ff68b-kube-api-access-s5g85\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.629789 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.629723 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.629858 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.629803 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.732242 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.732199 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g85\" (UniqueName: \"kubernetes.io/projected/68654e8e-63aa-4547-bd5c-11e0be2ff68b-kube-api-access-s5g85\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.732410 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.732284 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.732410 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.732323 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.733038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.733011 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.735245 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.735221 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.789451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.789383 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g85\" (UniqueName: \"kubernetes.io/projected/68654e8e-63aa-4547-bd5c-11e0be2ff68b-kube-api-access-s5g85\") pod \"maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.806376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.806352 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:13:57.998616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:57.998582 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr"] Nov 03 19:13:58.000254 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:13:58.000222 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e29b9f_c2a3_464e_9e89_22e3aefad77c.slice/crio-aed3986c9fb3517c9353fae39fa8eea4144acde99562da776df14c8d9fce2fac WatchSource:0}: Error finding container aed3986c9fb3517c9353fae39fa8eea4144acde99562da776df14c8d9fce2fac: Status 404 returned error can't find the container with id aed3986c9fb3517c9353fae39fa8eea4144acde99562da776df14c8d9fce2fac Nov 03 19:13:58.179245 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:58.179212 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj"] Nov 03 19:13:58.180542 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:13:58.180515 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68654e8e_63aa_4547_bd5c_11e0be2ff68b.slice/crio-4599920d64bac22f484f112799fe38faf13dc69b9885d8be1d4049e663ef4804 WatchSource:0}: Error finding container 4599920d64bac22f484f112799fe38faf13dc69b9885d8be1d4049e663ef4804: Status 404 returned error can't find the container with id 4599920d64bac22f484f112799fe38faf13dc69b9885d8be1d4049e663ef4804 Nov 03 19:13:58.351602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:58.351557 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" event={"ID":"68654e8e-63aa-4547-bd5c-11e0be2ff68b","Type":"ContainerStarted","Data":"4599920d64bac22f484f112799fe38faf13dc69b9885d8be1d4049e663ef4804"} Nov 03 19:13:58.353127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:58.353096 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" event={"ID":"a6e29b9f-c2a3-464e-9e89-22e3aefad77c","Type":"ContainerStarted","Data":"aed3986c9fb3517c9353fae39fa8eea4144acde99562da776df14c8d9fce2fac"} Nov 03 19:13:59.363075 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:59.363022 5988 generic.go:356] "Generic (PLEG): container finished" podID="68654e8e-63aa-4547-bd5c-11e0be2ff68b" containerID="812afa04d4172e421950c3c8f0e19cae4584db106b5634dd852c20f2890491ef" exitCode=0 Nov 03 19:13:59.363640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:59.363160 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" event={"ID":"68654e8e-63aa-4547-bd5c-11e0be2ff68b","Type":"ContainerDied","Data":"812afa04d4172e421950c3c8f0e19cae4584db106b5634dd852c20f2890491ef"} Nov 03 19:13:59.364896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:59.364872 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" event={"ID":"a6e29b9f-c2a3-464e-9e89-22e3aefad77c","Type":"ContainerStarted","Data":"a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709"} Nov 03 19:13:59.365054 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:59.364993 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:13:59.584366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:13:59.584304 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" podStartSLOduration=2.623471975 podStartE2EDuration="3.584284447s" totalImagesPullingTime="960.812472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:59.583207049 +0000 UTC m=+2114.464623152" watchObservedRunningTime="2026-11-03 19:13:59.584284447 +0000 UTC m=+2114.465700552" Nov 03 19:14:00.520076 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.520053 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:14:00.572907 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.572882 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68654e8e-63aa-4547-bd5c-11e0be2ff68b-kube-api-access-s5g85\" (UniqueName: \"kubernetes.io/projected/68654e8e-63aa-4547-bd5c-11e0be2ff68b-kube-api-access-s5g85\") pod \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " Nov 03 19:14:00.573055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.572930 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-openshift-ca\") pod \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " Nov 03 19:14:00.573128 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.573116 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-xks-ca\") pod \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\" (UID: \"68654e8e-63aa-4547-bd5c-11e0be2ff68b\") " Nov 03 19:14:00.573337 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.573312 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-openshift-ca" pod "68654e8e-63aa-4547-bd5c-11e0be2ff68b" (UID: "68654e8e-63aa-4547-bd5c-11e0be2ff68b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:00.573612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.573593 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:00.574953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.574934 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68654e8e-63aa-4547-bd5c-11e0be2ff68b-kube-api-access-s5g85" pod "68654e8e-63aa-4547-bd5c-11e0be2ff68b" (UID: "68654e8e-63aa-4547-bd5c-11e0be2ff68b"). InnerVolumeSpecName "kube-api-access-s5g85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:00.575068 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.575053 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-xks-ca" pod "68654e8e-63aa-4547-bd5c-11e0be2ff68b" (UID: "68654e8e-63aa-4547-bd5c-11e0be2ff68b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:00.675295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.675209 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68654e8e-63aa-4547-bd5c-11e0be2ff68b-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:00.675295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:00.675243 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5g85\" (UniqueName: \"kubernetes.io/projected/68654e8e-63aa-4547-bd5c-11e0be2ff68b-kube-api-access-s5g85\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:01.377894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:01.377854 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" Nov 03 19:14:01.378052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:01.377898 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj" event={"ID":"68654e8e-63aa-4547-bd5c-11e0be2ff68b","Type":"ContainerDied","Data":"4599920d64bac22f484f112799fe38faf13dc69b9885d8be1d4049e663ef4804"} Nov 03 19:14:01.378052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:01.377950 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4599920d64bac22f484f112799fe38faf13dc69b9885d8be1d4049e663ef4804" Nov 03 19:14:02.213441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:02.213396 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:05.375849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:05.375816 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:14:07.001775 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:07.001725 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:09.533265 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:09.533212 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:09.746935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:09.746907 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:11.928733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:11.928675 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:12.172076 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.172039 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d"] Nov 03 19:14:12.172742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.172727 5988 cpu_manager.go:414] "containerMap: removing container" podUID="68654e8e-63aa-4547-bd5c-11e0be2ff68b" containerName="revoke-keys" Nov 03 19:14:12.172802 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.172745 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68654e8e-63aa-4547-bd5c-11e0be2ff68b" containerName="revoke-keys" Nov 03 19:14:12.172876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.172866 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="68654e8e-63aa-4547-bd5c-11e0be2ff68b" containerName="revoke-keys" Nov 03 19:14:12.184613 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.184532 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.225698 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.225661 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.225847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.225785 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8pl9\" (UniqueName: \"kubernetes.io/projected/7fd330aa-0165-4f10-9afd-9a64421e60dc-kube-api-access-h8pl9\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.226020 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.226002 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.230367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.230331 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d"] Nov 03 19:14:12.233477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.233417 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:14:12.233639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.233619 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:14:12.328925 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.328888 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8pl9\" (UniqueName: \"kubernetes.io/projected/7fd330aa-0165-4f10-9afd-9a64421e60dc-kube-api-access-h8pl9\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.329225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.329194 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.329389 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.329271 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.330083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.330059 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.331650 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.331629 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.386133 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.386104 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8pl9\" (UniqueName: \"kubernetes.io/projected/7fd330aa-0165-4f10-9afd-9a64421e60dc-kube-api-access-h8pl9\") pod \"maas-api-revoke-keys-e2e-api-a-042f66-dvr8d\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.499083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.499012 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:12.869330 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:12.869305 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d"] Nov 03 19:14:12.871591 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:14:12.871561 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd330aa_0165_4f10_9afd_9a64421e60dc.slice/crio-3e94749775f9fb69e0606297d9930ff19cd14c293fcfc389972d01b63000b2b1 WatchSource:0}: Error finding container 3e94749775f9fb69e0606297d9930ff19cd14c293fcfc389972d01b63000b2b1: Status 404 returned error can't find the container with id 3e94749775f9fb69e0606297d9930ff19cd14c293fcfc389972d01b63000b2b1 Nov 03 19:14:13.465157 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:13.465116 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" event={"ID":"7fd330aa-0165-4f10-9afd-9a64421e60dc","Type":"ContainerStarted","Data":"0b1b76108268b15da0faa5920ead142f47c8c0781fc0e601d77cd6aa579cb741"} Nov 03 19:14:13.465554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:13.465176 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" event={"ID":"7fd330aa-0165-4f10-9afd-9a64421e60dc","Type":"ContainerStarted","Data":"3e94749775f9fb69e0606297d9930ff19cd14c293fcfc389972d01b63000b2b1"} Nov 03 19:14:13.575651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:13.575603 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" podStartSLOduration=1.5755881010000001 podStartE2EDuration="1.575588101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:13.573427503 +0000 UTC m=+2128.454843606" watchObservedRunningTime="2026-11-03 19:14:13.575588101 +0000 UTC m=+2128.457004202" Nov 03 19:14:14.472932 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:14.472901 5988 generic.go:356] "Generic (PLEG): container finished" podID="7fd330aa-0165-4f10-9afd-9a64421e60dc" containerID="0b1b76108268b15da0faa5920ead142f47c8c0781fc0e601d77cd6aa579cb741" exitCode=0 Nov 03 19:14:14.473340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:14.472988 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" event={"ID":"7fd330aa-0165-4f10-9afd-9a64421e60dc","Type":"ContainerDied","Data":"0b1b76108268b15da0faa5920ead142f47c8c0781fc0e601d77cd6aa579cb741"} Nov 03 19:14:15.637484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.637458 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:15.673193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.673168 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd330aa-0165-4f10-9afd-9a64421e60dc-kube-api-access-h8pl9\" (UniqueName: \"kubernetes.io/projected/7fd330aa-0165-4f10-9afd-9a64421e60dc-kube-api-access-h8pl9\") pod \"7fd330aa-0165-4f10-9afd-9a64421e60dc\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " Nov 03 19:14:15.673311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.673219 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-openshift-ca\") pod \"7fd330aa-0165-4f10-9afd-9a64421e60dc\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " Nov 03 19:14:15.673311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.673248 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-xks-ca\") pod \"7fd330aa-0165-4f10-9afd-9a64421e60dc\" (UID: \"7fd330aa-0165-4f10-9afd-9a64421e60dc\") " Nov 03 19:14:15.673596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.673565 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-openshift-ca" pod "7fd330aa-0165-4f10-9afd-9a64421e60dc" (UID: "7fd330aa-0165-4f10-9afd-9a64421e60dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:15.673896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.673875 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:15.675304 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.675283 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-xks-ca" pod "7fd330aa-0165-4f10-9afd-9a64421e60dc" (UID: "7fd330aa-0165-4f10-9afd-9a64421e60dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:15.675418 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.675388 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd330aa-0165-4f10-9afd-9a64421e60dc-kube-api-access-h8pl9" pod "7fd330aa-0165-4f10-9afd-9a64421e60dc" (UID: "7fd330aa-0165-4f10-9afd-9a64421e60dc"). InnerVolumeSpecName "kube-api-access-h8pl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:15.775006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.774927 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8pl9\" (UniqueName: \"kubernetes.io/projected/7fd330aa-0165-4f10-9afd-9a64421e60dc-kube-api-access-h8pl9\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:15.775006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:15.774956 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd330aa-0165-4f10-9afd-9a64421e60dc-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:16.487796 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:16.487766 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" Nov 03 19:14:16.487973 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:16.487772 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d" event={"ID":"7fd330aa-0165-4f10-9afd-9a64421e60dc","Type":"ContainerDied","Data":"3e94749775f9fb69e0606297d9930ff19cd14c293fcfc389972d01b63000b2b1"} Nov 03 19:14:16.487973 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:16.487894 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e94749775f9fb69e0606297d9930ff19cd14c293fcfc389972d01b63000b2b1" Nov 03 19:14:16.553583 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:16.553553 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:21.351658 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:21.351622 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:23.158641 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:23.158612 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t"] Nov 03 19:14:23.279577 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:23.279550 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f"] Nov 03 19:14:23.608276 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:23.608234 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:23.831475 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:23.831438 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:24.146464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.146422 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc"] Nov 03 19:14:24.147229 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.147205 5988 cpu_manager.go:414] "containerMap: removing container" podUID="7fd330aa-0165-4f10-9afd-9a64421e60dc" containerName="revoke-keys" Nov 03 19:14:24.147229 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.147224 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd330aa-0165-4f10-9afd-9a64421e60dc" containerName="revoke-keys" Nov 03 19:14:24.147406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.147344 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd330aa-0165-4f10-9afd-9a64421e60dc" containerName="revoke-keys" Nov 03 19:14:24.150944 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.150927 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.201751 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.201727 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:14:24.202180 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.201906 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:14:24.206359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.206335 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc"] Nov 03 19:14:24.279186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.279151 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5dt\" (UniqueName: \"kubernetes.io/projected/0f68ca1b-c585-4ecd-b10d-6471a30a7add-kube-api-access-tc5dt\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.279552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.279523 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.279736 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.279568 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.381408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.381378 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5dt\" (UniqueName: \"kubernetes.io/projected/0f68ca1b-c585-4ecd-b10d-6471a30a7add-kube-api-access-tc5dt\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.381606 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.381510 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.381606 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.381539 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.382317 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.382296 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.384221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.384194 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.438854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.438801 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5dt\" (UniqueName: \"kubernetes.io/projected/0f68ca1b-c585-4ecd-b10d-6471a30a7add-kube-api-access-tc5dt\") pod \"maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.464518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.464499 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:24.862119 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:24.862092 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc"] Nov 03 19:14:24.864984 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:14:24.864957 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f68ca1b_c585_4ecd_b10d_6471a30a7add.slice/crio-464365b39a9d584889f7dc5f71261907e0611f3eb592c3e8ffb494229150bc18 WatchSource:0}: Error finding container 464365b39a9d584889f7dc5f71261907e0611f3eb592c3e8ffb494229150bc18: Status 404 returned error can't find the container with id 464365b39a9d584889f7dc5f71261907e0611f3eb592c3e8ffb494229150bc18 Nov 03 19:14:25.559895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:25.559783 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" event={"ID":"0f68ca1b-c585-4ecd-b10d-6471a30a7add","Type":"ContainerStarted","Data":"0f19cd8a7952f3482bc699c3334cef70a4b55715d9f2f9e741eb0e99213b7dd2"} Nov 03 19:14:25.560339 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:25.559904 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" event={"ID":"0f68ca1b-c585-4ecd-b10d-6471a30a7add","Type":"ContainerStarted","Data":"464365b39a9d584889f7dc5f71261907e0611f3eb592c3e8ffb494229150bc18"} Nov 03 19:14:25.670928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:25.670879 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" podStartSLOduration=1.670865228 podStartE2EDuration="1.670865228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:25.667999095 +0000 UTC m=+2140.549415198" watchObservedRunningTime="2026-11-03 19:14:25.670865228 +0000 UTC m=+2140.552281330" Nov 03 19:14:25.937323 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:14:25.937286 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:14:26.567733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:26.567668 5988 generic.go:356] "Generic (PLEG): container finished" podID="0f68ca1b-c585-4ecd-b10d-6471a30a7add" containerID="0f19cd8a7952f3482bc699c3334cef70a4b55715d9f2f9e741eb0e99213b7dd2" exitCode=0 Nov 03 19:14:26.567733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:26.567716 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" event={"ID":"0f68ca1b-c585-4ecd-b10d-6471a30a7add","Type":"ContainerDied","Data":"0f19cd8a7952f3482bc699c3334cef70a4b55715d9f2f9e741eb0e99213b7dd2"} Nov 03 19:14:26.649991 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:26.649950 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:27.736595 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.736572 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:27.833879 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.833813 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-xks-ca\") pod \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " Nov 03 19:14:27.834013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.833936 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f68ca1b-c585-4ecd-b10d-6471a30a7add-kube-api-access-tc5dt\" (UniqueName: \"kubernetes.io/projected/0f68ca1b-c585-4ecd-b10d-6471a30a7add-kube-api-access-tc5dt\") pod \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " Nov 03 19:14:27.834013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.834006 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-openshift-ca\") pod \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\" (UID: \"0f68ca1b-c585-4ecd-b10d-6471a30a7add\") " Nov 03 19:14:27.834344 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.834318 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-openshift-ca" pod "0f68ca1b-c585-4ecd-b10d-6471a30a7add" (UID: "0f68ca1b-c585-4ecd-b10d-6471a30a7add"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:27.835851 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.835829 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-xks-ca" pod "0f68ca1b-c585-4ecd-b10d-6471a30a7add" (UID: "0f68ca1b-c585-4ecd-b10d-6471a30a7add"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:27.835955 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.835934 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f68ca1b-c585-4ecd-b10d-6471a30a7add-kube-api-access-tc5dt" pod "0f68ca1b-c585-4ecd-b10d-6471a30a7add" (UID: "0f68ca1b-c585-4ecd-b10d-6471a30a7add"). InnerVolumeSpecName "kube-api-access-tc5dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:27.935697 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.935663 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:27.935840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.935709 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5dt\" (UniqueName: \"kubernetes.io/projected/0f68ca1b-c585-4ecd-b10d-6471a30a7add-kube-api-access-tc5dt\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:27.935840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:27.935728 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f68ca1b-c585-4ecd-b10d-6471a30a7add-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:28.159969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.159885 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" podUID="aae6d193-38ce-41a5-9c1e-27a7996533ce" containerName="payload-processing" containerID="cri-o://8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266" gracePeriod=25 Nov 03 19:14:28.279874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.279837 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" podUID="6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" containerName="payload-pre-processing" containerID="cri-o://e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b" gracePeriod=25 Nov 03 19:14:28.431263 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.431233 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:14:28.545723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.544186 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae6d193-38ce-41a5-9c1e-27a7996533ce-kube-api-access-6tjcr\" (UniqueName: \"kubernetes.io/projected/aae6d193-38ce-41a5-9c1e-27a7996533ce-kube-api-access-6tjcr\") pod \"aae6d193-38ce-41a5-9c1e-27a7996533ce\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " Nov 03 19:14:28.545723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.544284 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae6d193-38ce-41a5-9c1e-27a7996533ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae6d193-38ce-41a5-9c1e-27a7996533ce-plugins-config-volume\") pod \"aae6d193-38ce-41a5-9c1e-27a7996533ce\" (UID: \"aae6d193-38ce-41a5-9c1e-27a7996533ce\") " Nov 03 19:14:28.545960 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.545764 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae6d193-38ce-41a5-9c1e-27a7996533ce-plugins-config-volume" pod "aae6d193-38ce-41a5-9c1e-27a7996533ce" (UID: "aae6d193-38ce-41a5-9c1e-27a7996533ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:28.547101 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.547076 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae6d193-38ce-41a5-9c1e-27a7996533ce-kube-api-access-6tjcr" pod "aae6d193-38ce-41a5-9c1e-27a7996533ce" (UID: "aae6d193-38ce-41a5-9c1e-27a7996533ce"). InnerVolumeSpecName "kube-api-access-6tjcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:28.560212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.560188 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:14:28.591059 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.591032 5988 generic.go:356] "Generic (PLEG): container finished" podID="6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" containerID="e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b" exitCode=0 Nov 03 19:14:28.591183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.591073 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" Nov 03 19:14:28.591183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.591110 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" event={"ID":"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40","Type":"ContainerDied","Data":"e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b"} Nov 03 19:14:28.591183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.591163 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f" event={"ID":"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40","Type":"ContainerDied","Data":"6dd152b15baf3a60625114ac94f4f793f1a5f528ae37fcd71fec404fe7b908d4"} Nov 03 19:14:28.591334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.591187 5988 scope.go:122] "RemoveContainer" containerID="e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b" Nov 03 19:14:28.592478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.592430 5988 generic.go:356] "Generic (PLEG): container finished" podID="aae6d193-38ce-41a5-9c1e-27a7996533ce" containerID="8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266" exitCode=0 Nov 03 19:14:28.592478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.592470 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" Nov 03 19:14:28.592567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.592508 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" event={"ID":"aae6d193-38ce-41a5-9c1e-27a7996533ce","Type":"ContainerDied","Data":"8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266"} Nov 03 19:14:28.592567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.592549 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t" event={"ID":"aae6d193-38ce-41a5-9c1e-27a7996533ce","Type":"ContainerDied","Data":"860bdeec3682e6a736fe547a0a9589960163ed29901cae1cd6097913fcf9b002"} Nov 03 19:14:28.594059 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.594042 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" Nov 03 19:14:28.594179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.594061 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc" event={"ID":"0f68ca1b-c585-4ecd-b10d-6471a30a7add","Type":"ContainerDied","Data":"464365b39a9d584889f7dc5f71261907e0611f3eb592c3e8ffb494229150bc18"} Nov 03 19:14:28.594179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.594103 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="464365b39a9d584889f7dc5f71261907e0611f3eb592c3e8ffb494229150bc18" Nov 03 19:14:28.601787 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.601710 5988 scope.go:122] "RemoveContainer" containerID="e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b" Nov 03 19:14:28.602001 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:14:28.601982 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b\": container with ID starting with e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b not found: ID does not exist" containerID="e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b" Nov 03 19:14:28.602071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.602012 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b"} err="failed to get container status \"e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b\": rpc error: code = NotFound desc = could not find container \"e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b\": container with ID starting with e81c6788182c4f087eff9c3bf50dcb58de0df3fa51b641ad067919a98562a00b not found: ID does not exist" Nov 03 19:14:28.602071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.602037 5988 scope.go:122] "RemoveContainer" containerID="8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266" Nov 03 19:14:28.611746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.611729 5988 scope.go:122] "RemoveContainer" containerID="8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266" Nov 03 19:14:28.612009 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:14:28.611992 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266\": container with ID starting with 8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266 not found: ID does not exist" containerID="8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266" Nov 03 19:14:28.612061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.612015 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266"} err="failed to get container status \"8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266\": rpc error: code = NotFound desc = could not find container \"8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266\": container with ID starting with 8c3678638aa08899d8c15e33f70929c05413bc031e080d9c54dae081b91bd266 not found: ID does not exist" Nov 03 19:14:28.645861 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.645840 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-kube-api-access-66gsp\" (UniqueName: \"kubernetes.io/projected/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-kube-api-access-66gsp\") pod \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " Nov 03 19:14:28.646013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.646002 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-plugins-config-volume\") pod \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\" (UID: \"6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40\") " Nov 03 19:14:28.646337 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.646319 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-plugins-config-volume" pod "6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" (UID: "6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:28.646395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.646382 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tjcr\" (UniqueName: \"kubernetes.io/projected/aae6d193-38ce-41a5-9c1e-27a7996533ce-kube-api-access-6tjcr\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:28.646433 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.646398 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aae6d193-38ce-41a5-9c1e-27a7996533ce-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:28.647820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.647799 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-kube-api-access-66gsp" pod "6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" (UID: "6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40"). InnerVolumeSpecName "kube-api-access-66gsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:28.747628 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.747603 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:28.747628 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.747627 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66gsp\" (UniqueName: \"kubernetes.io/projected/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40-kube-api-access-66gsp\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:28.774405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.774381 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t"] Nov 03 19:14:28.779941 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:28.779918 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-042f66-6f8988d659-hhk2t"] Nov 03 19:14:29.007362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:29.007289 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f"] Nov 03 19:14:29.021627 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:29.021596 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-042f66-954b7d67f-8fk7f"] Nov 03 19:14:29.274404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:29.274332 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" path="/var/lib/kubelet/pods/6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40/volumes" Nov 03 19:14:29.274846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:29.274828 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae6d193-38ce-41a5-9c1e-27a7996533ce" path="/var/lib/kubelet/pods/aae6d193-38ce-41a5-9c1e-27a7996533ce/volumes" Nov 03 19:14:31.485032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:31.484950 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:32.267385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:32.267353 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:35.751236 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:35.751196 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:36.283409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:36.283379 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:36.493369 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:36.493335 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:37.243745 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:37.243711 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:14:38.036409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:38.036369 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:38.246195 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:38.246164 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:38.487352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:38.487317 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:38.575482 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:38.575452 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d"] Nov 03 19:14:38.621676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:38.621650 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-042f66-dvr8d"] Nov 03 19:14:38.999791 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:38.999748 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:39.314160 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:39.314070 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd330aa-0165-4f10-9afd-9a64421e60dc" path="/var/lib/kubelet/pods/7fd330aa-0165-4f10-9afd-9a64421e60dc/volumes" Nov 03 19:14:43.151390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:43.151347 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:45.033423 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:45.033388 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:45.249583 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:45.249544 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:45.515523 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:45.515475 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:47.283635 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:47.283554 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:48.155520 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.155484 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr"] Nov 03 19:14:48.155753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.155715 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" podUID="6f095571-f9ab-4d4e-8dcb-72b49195e131" containerName="maas-api" containerID="cri-o://6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e" gracePeriod=30 Nov 03 19:14:48.314630 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.314587 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:48.501370 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.501337 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:14:48.604726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.601211 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:48.668915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.665920 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-metrics-certs\") pod \"6f095571-f9ab-4d4e-8dcb-72b49195e131\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " Nov 03 19:14:48.668915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.666090 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls\") pod \"6f095571-f9ab-4d4e-8dcb-72b49195e131\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " Nov 03 19:14:48.668915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.666145 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f095571-f9ab-4d4e-8dcb-72b49195e131-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f095571-f9ab-4d4e-8dcb-72b49195e131-openshift-service-ca\") pod \"6f095571-f9ab-4d4e-8dcb-72b49195e131\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " Nov 03 19:14:48.668915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.666224 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f095571-f9ab-4d4e-8dcb-72b49195e131-kube-api-access-ndzhk\" (UniqueName: \"kubernetes.io/projected/6f095571-f9ab-4d4e-8dcb-72b49195e131-kube-api-access-ndzhk\") pod \"6f095571-f9ab-4d4e-8dcb-72b49195e131\" (UID: \"6f095571-f9ab-4d4e-8dcb-72b49195e131\") " Nov 03 19:14:48.668915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.667695 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f095571-f9ab-4d4e-8dcb-72b49195e131-openshift-service-ca" pod "6f095571-f9ab-4d4e-8dcb-72b49195e131" (UID: "6f095571-f9ab-4d4e-8dcb-72b49195e131"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:14:48.669344 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.669302 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-metrics-certs" pod "6f095571-f9ab-4d4e-8dcb-72b49195e131" (UID: "6f095571-f9ab-4d4e-8dcb-72b49195e131"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:48.687735 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.687163 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f095571-f9ab-4d4e-8dcb-72b49195e131-kube-api-access-ndzhk" pod "6f095571-f9ab-4d4e-8dcb-72b49195e131" (UID: "6f095571-f9ab-4d4e-8dcb-72b49195e131"). InnerVolumeSpecName "kube-api-access-ndzhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:14:48.689041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.689007 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls" pod "6f095571-f9ab-4d4e-8dcb-72b49195e131" (UID: "6f095571-f9ab-4d4e-8dcb-72b49195e131"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:14:48.742586 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.742469 5988 generic.go:356] "Generic (PLEG): container finished" podID="6f095571-f9ab-4d4e-8dcb-72b49195e131" containerID="6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e" exitCode=0 Nov 03 19:14:48.742586 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.742520 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" Nov 03 19:14:48.742586 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.742560 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" event={"ID":"6f095571-f9ab-4d4e-8dcb-72b49195e131","Type":"ContainerDied","Data":"6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e"} Nov 03 19:14:48.742888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.742621 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr" event={"ID":"6f095571-f9ab-4d4e-8dcb-72b49195e131","Type":"ContainerDied","Data":"a97c9b444ce75d5732eff8ce42321494865ca746b03243cdc685f5ecb1d54111"} Nov 03 19:14:48.742888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.742649 5988 scope.go:122] "RemoveContainer" containerID="6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e" Nov 03 19:14:48.768056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.766475 5988 scope.go:122] "RemoveContainer" containerID="6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e" Nov 03 19:14:48.768056 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:14:48.766926 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e\": container with ID starting with 6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e not found: ID does not exist" containerID="6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e" Nov 03 19:14:48.768056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.766966 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e"} err="failed to get container status \"6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e\": rpc error: code = NotFound desc = could not find container \"6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e\": container with ID starting with 6319e9149be39c730b3187a7ded6265eedc8200676b99bc68757ffc48eaf790e not found: ID does not exist" Nov 03 19:14:48.768056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.767962 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:48.768056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.767981 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f095571-f9ab-4d4e-8dcb-72b49195e131-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:48.768056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.768000 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f095571-f9ab-4d4e-8dcb-72b49195e131-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:48.768056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.768009 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndzhk\" (UniqueName: \"kubernetes.io/projected/6f095571-f9ab-4d4e-8dcb-72b49195e131-kube-api-access-ndzhk\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:14:48.866714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.866657 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:48.898333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.898298 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr"] Nov 03 19:14:48.902796 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:48.902768 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-bf3e1c-58757796b9-2l7pr"] Nov 03 19:14:49.073548 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.073511 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:49.137268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.137235 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl"] Nov 03 19:14:49.137979 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.137956 5988 cpu_manager.go:414] "containerMap: removing container" podUID="6f095571-f9ab-4d4e-8dcb-72b49195e131" containerName="maas-api" Nov 03 19:14:49.138113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.137978 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f095571-f9ab-4d4e-8dcb-72b49195e131" containerName="maas-api" Nov 03 19:14:49.138113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138010 5988 cpu_manager.go:414] "containerMap: removing container" podUID="0f68ca1b-c585-4ecd-b10d-6471a30a7add" containerName="revoke-keys" Nov 03 19:14:49.138113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138017 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f68ca1b-c585-4ecd-b10d-6471a30a7add" containerName="revoke-keys" Nov 03 19:14:49.138113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138073 5988 cpu_manager.go:414] "containerMap: removing container" podUID="aae6d193-38ce-41a5-9c1e-27a7996533ce" containerName="payload-processing" Nov 03 19:14:49.138113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138081 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae6d193-38ce-41a5-9c1e-27a7996533ce" containerName="payload-processing" Nov 03 19:14:49.138113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138091 5988 cpu_manager.go:414] "containerMap: removing container" podUID="6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" containerName="payload-pre-processing" Nov 03 19:14:49.138113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138098 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" containerName="payload-pre-processing" Nov 03 19:14:49.138467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138242 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f095571-f9ab-4d4e-8dcb-72b49195e131" containerName="maas-api" Nov 03 19:14:49.138467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138259 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f68ca1b-c585-4ecd-b10d-6471a30a7add" containerName="revoke-keys" Nov 03 19:14:49.138467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138279 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae6d193-38ce-41a5-9c1e-27a7996533ce" containerName="payload-processing" Nov 03 19:14:49.138467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.138287 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c44a606-8d5c-4f14-b77d-b5c4c0ca2c40" containerName="payload-pre-processing" Nov 03 19:14:49.143116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.143093 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.146823 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.146800 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc"] Nov 03 19:14:49.152602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.152579 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4320e5-dk5kc"] Nov 03 19:14:49.192945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.192910 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9dd9f2-openshift-default-dockercfg-sgk7t\"" Nov 03 19:14:49.203399 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.203359 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl"] Nov 03 19:14:49.213041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.213007 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-certs\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.213306 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.213283 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-credential-socket\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.213523 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.213502 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-data\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.213919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.213883 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-socket\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.214186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.214157 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-token\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.214365 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.214342 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmkr\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-kube-api-access-5tmkr\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.214592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.214573 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-podinfo\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.216519 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.214871 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-envoy\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.216519 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.214944 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istiod-ca-cert\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.274128 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.274087 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f68ca1b-c585-4ecd-b10d-6471a30a7add" path="/var/lib/kubelet/pods/0f68ca1b-c585-4ecd-b10d-6471a30a7add/volumes" Nov 03 19:14:49.274731 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.274668 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f095571-f9ab-4d4e-8dcb-72b49195e131" path="/var/lib/kubelet/pods/6f095571-f9ab-4d4e-8dcb-72b49195e131/volumes" Nov 03 19:14:49.312572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.312507 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:49.320573 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.320533 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmkr\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-kube-api-access-5tmkr\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.321184 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.321159 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-podinfo\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.321415 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.321398 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-envoy\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.321557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.321542 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istiod-ca-cert\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.321742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.321727 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-certs\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.321879 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.321865 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-credential-socket\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.330963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.330877 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-data\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.331250 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.331230 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-socket\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.331390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.331374 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-token\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.333043 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.333016 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-credential-socket\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.370708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.342529 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-data\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.370708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.342529 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istiod-ca-cert\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.370708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.343056 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-socket\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.370708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.343295 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-certs\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.370708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.350305 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-podinfo\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.401615 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.401585 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-envoy\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.402573 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.402542 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmkr\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-kube-api-access-5tmkr\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.464765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.464722 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-token\") pod \"e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.465787 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.465762 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:49.523065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.523030 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:49.739376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.739272 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:49.912732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.912396 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl"] Nov 03 19:14:49.915525 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:14:49.915492 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807178f7_57ea_44bc_94ed_56e4f3ea3cae.slice/crio-aa96bd7bd9dfc21a6c95e865785282297b31a8688ad10c17357c6fb93b7da150 WatchSource:0}: Error finding container aa96bd7bd9dfc21a6c95e865785282297b31a8688ad10c17357c6fb93b7da150: Status 404 returned error can't find the container with id aa96bd7bd9dfc21a6c95e865785282297b31a8688ad10c17357c6fb93b7da150 Nov 03 19:14:49.917977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.917943 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:14:49.918079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.918011 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:14:49.918079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:49.918042 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:14:50.093536 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.093501 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:50.512640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.512600 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:50.593761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.593718 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj"] Nov 03 19:14:50.597283 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.597249 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-bf3e1c-gnbgj"] Nov 03 19:14:50.727422 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.727386 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:50.758749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.758708 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" event={"ID":"807178f7-57ea-44bc-94ed-56e4f3ea3cae","Type":"ContainerStarted","Data":"49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f"} Nov 03 19:14:50.758924 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.758759 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" event={"ID":"807178f7-57ea-44bc-94ed-56e4f3ea3cae","Type":"ContainerStarted","Data":"aa96bd7bd9dfc21a6c95e865785282297b31a8688ad10c17357c6fb93b7da150"} Nov 03 19:14:50.874359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:50.874295 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podStartSLOduration=1.8742764250000001 podStartE2EDuration="1.874276425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:50.871947691 +0000 UTC m=+2165.753363819" watchObservedRunningTime="2026-11-03 19:14:50.874276425 +0000 UTC m=+2165.755692528" Nov 03 19:14:51.076970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:51.076935 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:51.275233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:51.275153 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfe2657-334b-4267-aa77-4a284674c162" path="/var/lib/kubelet/pods/fbfe2657-334b-4267-aa77-4a284674c162/volumes" Nov 03 19:14:51.466219 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:51.466178 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:51.472604 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:51.472573 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:51.764958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:51.764924 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:51.766005 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:51.765985 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:14:53.420777 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:53.420740 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:53.745885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:53.745796 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:53.956497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:53.956403 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:55.429400 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:55.429362 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:55.939979 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:14:55.939940 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:14:56.402338 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:56.402303 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:56.756150 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:56.756059 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj"] Nov 03 19:14:56.756590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:56.756285 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" containerID="cri-o://6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e" gracePeriod=30 Nov 03 19:14:56.771792 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:56.771760 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:56.981430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:56.981394 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:57.185489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.185452 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:57.361902 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.361859 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t"] Nov 03 19:14:57.371280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.371243 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.413399 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.413344 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t"] Nov 03 19:14:57.420837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.420793 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9dd9f2-dockercfg-tpz6c\"" Nov 03 19:14:57.420837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.420836 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9dd9f2\"" Nov 03 19:14:57.459054 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.458962 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt25c\" (UniqueName: \"kubernetes.io/projected/ba5f3c45-1e8f-4e40-862a-73f55de550ea-kube-api-access-dt25c\") pod \"payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.459281 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.459251 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5f3c45-1e8f-4e40-862a-73f55de550ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.483323 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.483283 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s"] Nov 03 19:14:57.488835 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.488806 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:57.544866 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.543957 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s"] Nov 03 19:14:57.565641 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.565579 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5f3c45-1e8f-4e40-862a-73f55de550ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.565960 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.565787 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt25c\" (UniqueName: \"kubernetes.io/projected/ba5f3c45-1e8f-4e40-862a-73f55de550ea-kube-api-access-dt25c\") pod \"payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.567564 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.567539 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5f3c45-1e8f-4e40-862a-73f55de550ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.629461 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.629422 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt25c\" (UniqueName: \"kubernetes.io/projected/ba5f3c45-1e8f-4e40-862a-73f55de550ea-kube-api-access-dt25c\") pod \"payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.667796 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.667752 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9hpz\" (UniqueName: \"kubernetes.io/projected/e56dd326-b071-4d84-8d5a-352173b3504d-kube-api-access-w9hpz\") pod \"payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:57.668275 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.668241 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56dd326-b071-4d84-8d5a-352173b3504d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:57.689542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.689500 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:57.771404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.769527 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56dd326-b071-4d84-8d5a-352173b3504d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:57.771404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.769736 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9hpz\" (UniqueName: \"kubernetes.io/projected/e56dd326-b071-4d84-8d5a-352173b3504d-kube-api-access-w9hpz\") pod \"payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:57.771404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.771310 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56dd326-b071-4d84-8d5a-352173b3504d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:57.828034 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:57.828002 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9hpz\" (UniqueName: \"kubernetes.io/projected/e56dd326-b071-4d84-8d5a-352173b3504d-kube-api-access-w9hpz\") pod \"payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:58.068618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.068581 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t"] Nov 03 19:14:58.070350 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:14:58.070317 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5f3c45_1e8f_4e40_862a_73f55de550ea.slice/crio-f1a155d1960054cd796840e24dd127a2b541c0b6da2e653d8e22f82b6462ed0b WatchSource:0}: Error finding container f1a155d1960054cd796840e24dd127a2b541c0b6da2e653d8e22f82b6462ed0b: Status 404 returned error can't find the container with id f1a155d1960054cd796840e24dd127a2b541c0b6da2e653d8e22f82b6462ed0b Nov 03 19:14:58.107300 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.107268 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:58.317379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.317285 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:58.484808 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.484780 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s"] Nov 03 19:14:58.487987 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:14:58.487958 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56dd326_b071_4d84_8d5a_352173b3504d.slice/crio-0fa878841e89f44cf7b919bb0c8b34d91dbe88ec8d86b5abab7fab8158373550 WatchSource:0}: Error finding container 0fa878841e89f44cf7b919bb0c8b34d91dbe88ec8d86b5abab7fab8158373550: Status 404 returned error can't find the container with id 0fa878841e89f44cf7b919bb0c8b34d91dbe88ec8d86b5abab7fab8158373550 Nov 03 19:14:58.530375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.530105 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:58.776316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.776286 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:58.820307 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.820271 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" event={"ID":"ba5f3c45-1e8f-4e40-862a-73f55de550ea","Type":"ContainerStarted","Data":"ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e"} Nov 03 19:14:58.820504 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.820329 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" event={"ID":"ba5f3c45-1e8f-4e40-862a-73f55de550ea","Type":"ContainerStarted","Data":"f1a155d1960054cd796840e24dd127a2b541c0b6da2e653d8e22f82b6462ed0b"} Nov 03 19:14:58.820504 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.820372 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:14:58.822201 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.822178 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" event={"ID":"e56dd326-b071-4d84-8d5a-352173b3504d","Type":"ContainerStarted","Data":"0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3"} Nov 03 19:14:58.822333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.822217 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" event={"ID":"e56dd326-b071-4d84-8d5a-352173b3504d","Type":"ContainerStarted","Data":"0fa878841e89f44cf7b919bb0c8b34d91dbe88ec8d86b5abab7fab8158373550"} Nov 03 19:14:58.822333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.822245 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:14:58.929548 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.929418 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" podStartSLOduration=1.929400389 podStartE2EDuration="1.929400389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:58.929024218 +0000 UTC m=+2173.810440320" watchObservedRunningTime="2026-11-03 19:14:58.929400389 +0000 UTC m=+2173.810816495" Nov 03 19:14:58.980244 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:58.980212 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:14:59.041809 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.041747 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" podStartSLOduration=2.041726608 podStartE2EDuration="2.041726608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:59.038193703 +0000 UTC m=+2173.919609808" watchObservedRunningTime="2026-11-03 19:14:59.041726608 +0000 UTC m=+2173.923142715" Nov 03 19:14:59.239477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.239389 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5"] Nov 03 19:14:59.243930 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.243905 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.299411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.299377 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:14:59.299885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.299830 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:14:59.300155 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.300125 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5"] Nov 03 19:14:59.402919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.402881 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.403116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.403094 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshqk\" (UniqueName: \"kubernetes.io/projected/060fc8f2-76ef-4b86-82bc-0684bc7116aa-kube-api-access-pshqk\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.403215 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.403140 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.505116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.505009 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshqk\" (UniqueName: \"kubernetes.io/projected/060fc8f2-76ef-4b86-82bc-0684bc7116aa-kube-api-access-pshqk\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.505116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.505079 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.505367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.505261 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.505952 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.505930 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.507502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.507484 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.565498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.565459 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshqk\" (UniqueName: \"kubernetes.io/projected/060fc8f2-76ef-4b86-82bc-0684bc7116aa-kube-api-access-pshqk\") pod \"maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:14:59.859342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:14:59.859304 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:15:00.161833 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.161745 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh"] Nov 03 19:15:00.167094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.167070 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr"] Nov 03 19:15:00.167243 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.167195 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.171990 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.171968 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8"] Nov 03 19:15:00.172104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.172083 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.175872 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.175852 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.200017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.199988 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk"] Nov 03 19:15:00.204415 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.204393 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.205861 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.205835 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm"] Nov 03 19:15:00.217812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.214854 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.232378 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.232349 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv"] Nov 03 19:15:00.236549 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.236527 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.245312 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.245284 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj"] Nov 03 19:15:00.250250 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.250209 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.253331 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.253308 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9"] Nov 03 19:15:00.257544 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.257523 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.262831 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.262806 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh"] Nov 03 19:15:00.265929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.265907 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr"] Nov 03 19:15:00.268832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.268811 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8"] Nov 03 19:15:00.271283 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.271265 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5"] Nov 03 19:15:00.280993 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:00.280966 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060fc8f2_76ef_4b86_82bc_0684bc7116aa.slice/crio-1ae06ee589c2c3e1707c07583fef2cfd7e3cfd4d54cb78764d6466e3e7fedf2a WatchSource:0}: Error finding container 1ae06ee589c2c3e1707c07583fef2cfd7e3cfd4d54cb78764d6466e3e7fedf2a: Status 404 returned error can't find the container with id 1ae06ee589c2c3e1707c07583fef2cfd7e3cfd4d54cb78764d6466e3e7fedf2a Nov 03 19:15:00.315105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.314880 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm"] Nov 03 19:15:00.319074 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319037 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/344a6ecd-90de-4398-abff-9e246298d785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.319216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319197 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp5l\" (UniqueName: \"kubernetes.io/projected/344a6ecd-90de-4398-abff-9e246298d785-kube-api-access-pmp5l\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.319287 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319248 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkndt\" (UniqueName: \"kubernetes.io/projected/453e6d0d-1386-4605-90e8-428474307891-kube-api-access-pkndt\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.319416 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319390 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftm7\" (UniqueName: \"kubernetes.io/projected/5cffe7d7-51e0-49f6-9acf-2cef2889878f-kube-api-access-tftm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.319509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319494 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvms9\" (UniqueName: \"kubernetes.io/projected/34f3e8db-5cf4-4793-8771-05c8075a48f6-kube-api-access-xvms9\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.319592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319575 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjr6\" (UniqueName: \"kubernetes.io/projected/0664c8b7-6cd4-443a-9802-a7fecc7cd186-kube-api-access-fkjr6\") pod \"maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.319651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319610 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f3e8db-5cf4-4793-8771-05c8075a48f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.319747 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319654 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453e6d0d-1386-4605-90e8-428474307891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.319815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319784 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cffe7d7-51e0-49f6-9acf-2cef2889878f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.319875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.319828 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0664c8b7-6cd4-443a-9802-a7fecc7cd186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.365467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.365432 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9"] Nov 03 19:15:00.370937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.370903 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj"] Nov 03 19:15:00.375846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.375815 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk"] Nov 03 19:15:00.414270 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.414128 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv"] Nov 03 19:15:00.421897 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.421842 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftm7\" (UniqueName: \"kubernetes.io/projected/5cffe7d7-51e0-49f6-9acf-2cef2889878f-kube-api-access-tftm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.422078 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.421949 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eedf97f-8b5d-4c39-b931-76086a20b7fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.422078 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.421984 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvms9\" (UniqueName: \"kubernetes.io/projected/34f3e8db-5cf4-4793-8771-05c8075a48f6-kube-api-access-xvms9\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422627 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87f2l\" (UniqueName: \"kubernetes.io/projected/3eedf97f-8b5d-4c39-b931-76086a20b7fa-kube-api-access-87f2l\") pod \"maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422721 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjr6\" (UniqueName: \"kubernetes.io/projected/0664c8b7-6cd4-443a-9802-a7fecc7cd186-kube-api-access-fkjr6\") pod \"maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422753 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f3e8db-5cf4-4793-8771-05c8075a48f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422802 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453e6d0d-1386-4605-90e8-428474307891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422880 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f32a5c-5c74-4495-8d14-3bfecfe81596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422924 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cffe7d7-51e0-49f6-9acf-2cef2889878f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422959 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tns8g\" (UniqueName: \"kubernetes.io/projected/31e6242f-3785-4ac5-818c-20f85285bc49-kube-api-access-tns8g\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.422994 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0664c8b7-6cd4-443a-9802-a7fecc7cd186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.423095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.423078 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e6242f-3785-4ac5-818c-20f85285bc49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.425192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.423135 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/344a6ecd-90de-4398-abff-9e246298d785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.425192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.423159 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98bq\" (UniqueName: \"kubernetes.io/projected/81f32a5c-5c74-4495-8d14-3bfecfe81596-kube-api-access-l98bq\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.425192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.423240 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp5l\" (UniqueName: \"kubernetes.io/projected/344a6ecd-90de-4398-abff-9e246298d785-kube-api-access-pmp5l\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.425192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.423288 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkndt\" (UniqueName: \"kubernetes.io/projected/453e6d0d-1386-4605-90e8-428474307891-kube-api-access-pkndt\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.425192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.424561 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f3e8db-5cf4-4793-8771-05c8075a48f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.425753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.425729 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453e6d0d-1386-4605-90e8-428474307891-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.426318 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.426293 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0664c8b7-6cd4-443a-9802-a7fecc7cd186-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.426404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.426293 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/344a6ecd-90de-4398-abff-9e246298d785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.426543 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.426518 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cffe7d7-51e0-49f6-9acf-2cef2889878f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.479935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.479899 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvms9\" (UniqueName: \"kubernetes.io/projected/34f3e8db-5cf4-4793-8771-05c8075a48f6-kube-api-access-xvms9\") pod \"maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.480118 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.479991 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftm7\" (UniqueName: \"kubernetes.io/projected/5cffe7d7-51e0-49f6-9acf-2cef2889878f-kube-api-access-tftm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.480359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.480339 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkndt\" (UniqueName: \"kubernetes.io/projected/453e6d0d-1386-4605-90e8-428474307891-kube-api-access-pkndt\") pod \"maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.480433 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.480414 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjr6\" (UniqueName: \"kubernetes.io/projected/0664c8b7-6cd4-443a-9802-a7fecc7cd186-kube-api-access-fkjr6\") pod \"maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.482253 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.482229 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp5l\" (UniqueName: \"kubernetes.io/projected/344a6ecd-90de-4398-abff-9e246298d785-kube-api-access-pmp5l\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.484155 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.484136 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:00.493997 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.493926 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:00.503851 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.503824 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:00.525207 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.525169 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e6242f-3785-4ac5-818c-20f85285bc49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.525385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.525253 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98bq\" (UniqueName: \"kubernetes.io/projected/81f32a5c-5c74-4495-8d14-3bfecfe81596-kube-api-access-l98bq\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.525460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.525425 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eedf97f-8b5d-4c39-b931-76086a20b7fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.525517 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.525475 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87f2l\" (UniqueName: \"kubernetes.io/projected/3eedf97f-8b5d-4c39-b931-76086a20b7fa-kube-api-access-87f2l\") pod \"maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.525635 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.525614 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f32a5c-5c74-4495-8d14-3bfecfe81596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.525761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.525670 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns8g\" (UniqueName: \"kubernetes.io/projected/31e6242f-3785-4ac5-818c-20f85285bc49-kube-api-access-tns8g\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.525974 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.525950 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e6242f-3785-4ac5-818c-20f85285bc49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.526293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.526270 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eedf97f-8b5d-4c39-b931-76086a20b7fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.526419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.526404 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:00.526709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.526657 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f32a5c-5c74-4495-8d14-3bfecfe81596-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.536892 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.536858 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:00.588502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.587421 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98bq\" (UniqueName: \"kubernetes.io/projected/81f32a5c-5c74-4495-8d14-3bfecfe81596-kube-api-access-l98bq\") pod \"maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.613469 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.613430 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87f2l\" (UniqueName: \"kubernetes.io/projected/3eedf97f-8b5d-4c39-b931-76086a20b7fa-kube-api-access-87f2l\") pod \"maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.621637 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.621412 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tns8g\" (UniqueName: \"kubernetes.io/projected/31e6242f-3785-4ac5-818c-20f85285bc49-kube-api-access-tns8g\") pod \"maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.777973 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.777937 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:00.839285 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.839242 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" event={"ID":"060fc8f2-76ef-4b86-82bc-0684bc7116aa","Type":"ContainerStarted","Data":"acf8a5460069dc349bb541f813d03a485d7df96c028f411782574bbe81a9b325"} Nov 03 19:15:00.839498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.839321 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" event={"ID":"060fc8f2-76ef-4b86-82bc-0684bc7116aa","Type":"ContainerStarted","Data":"1ae06ee589c2c3e1707c07583fef2cfd7e3cfd4d54cb78764d6466e3e7fedf2a"} Nov 03 19:15:00.852146 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.852113 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:00.866610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.866581 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:00.872482 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.872455 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh"] Nov 03 19:15:00.873937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.873914 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:00.915206 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.915180 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8"] Nov 03 19:15:00.920277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:00.920249 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr"] Nov 03 19:15:00.930335 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:00.930145 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453e6d0d_1386_4605_90e8_428474307891.slice/crio-31a2948ce06ce3a26fe4c010e6f17c61df0f8725b06968a005402c8a04662421 WatchSource:0}: Error finding container 31a2948ce06ce3a26fe4c010e6f17c61df0f8725b06968a005402c8a04662421: Status 404 returned error can't find the container with id 31a2948ce06ce3a26fe4c010e6f17c61df0f8725b06968a005402c8a04662421 Nov 03 19:15:01.031095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.030796 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm"] Nov 03 19:15:01.033968 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:01.033931 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344a6ecd_90de_4398_abff_9e246298d785.slice/crio-b68eb2fb276cb8b8126878e3fbbc0f299a2eadd092e71fad1d7b663a85d65ecf WatchSource:0}: Error finding container b68eb2fb276cb8b8126878e3fbbc0f299a2eadd092e71fad1d7b663a85d65ecf: Status 404 returned error can't find the container with id b68eb2fb276cb8b8126878e3fbbc0f299a2eadd092e71fad1d7b663a85d65ecf Nov 03 19:15:01.075524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.075496 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk"] Nov 03 19:15:01.151432 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:01.151401 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0664c8b7_6cd4_443a_9802_a7fecc7cd186.slice/crio-7c2129ac38aec0777a94327671305d20c9bbb387ee688db3e047918dade83e6a WatchSource:0}: Error finding container 7c2129ac38aec0777a94327671305d20c9bbb387ee688db3e047918dade83e6a: Status 404 returned error can't find the container with id 7c2129ac38aec0777a94327671305d20c9bbb387ee688db3e047918dade83e6a Nov 03 19:15:01.165912 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.165854 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" podStartSLOduration=2.165835618 podStartE2EDuration="2.165835618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:01.163314246 +0000 UTC m=+2176.044730347" watchObservedRunningTime="2026-11-03 19:15:01.165835618 +0000 UTC m=+2176.047251725" Nov 03 19:15:01.249676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.249594 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv"] Nov 03 19:15:01.276929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.276904 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj"] Nov 03 19:15:01.284821 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.284800 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9"] Nov 03 19:15:01.352124 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:01.352093 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eedf97f_8b5d_4c39_b931_76086a20b7fa.slice/crio-3d771c79ee7712002794ceff270183c98c809921dc2f1cec2a9bfa92838ca839 WatchSource:0}: Error finding container 3d771c79ee7712002794ceff270183c98c809921dc2f1cec2a9bfa92838ca839: Status 404 returned error can't find the container with id 3d771c79ee7712002794ceff270183c98c809921dc2f1cec2a9bfa92838ca839 Nov 03 19:15:01.352374 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:01.352353 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e6242f_3785_4ac5_818c_20f85285bc49.slice/crio-3b4a5d95a11d428b674f9bca4183f7d6c8cf6b15599832f2e2782fa3500d4fe9 WatchSource:0}: Error finding container 3b4a5d95a11d428b674f9bca4183f7d6c8cf6b15599832f2e2782fa3500d4fe9: Status 404 returned error can't find the container with id 3b4a5d95a11d428b674f9bca4183f7d6c8cf6b15599832f2e2782fa3500d4fe9 Nov 03 19:15:01.449271 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:01.449205 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f32a5c_5c74_4495_8d14_3bfecfe81596.slice/crio-57c17080fca5745f87a00130e730c249af394325b55fb98e828b76e271847bab WatchSource:0}: Error finding container 57c17080fca5745f87a00130e730c249af394325b55fb98e828b76e271847bab: Status 404 returned error can't find the container with id 57c17080fca5745f87a00130e730c249af394325b55fb98e828b76e271847bab Nov 03 19:15:01.849905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.849872 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerStarted","Data":"8ebabb193c5b41626ff07ff0b87ac674826da8d4c8ce36d229cad90e7c7f11c3"} Nov 03 19:15:01.850093 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.849935 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerStarted","Data":"8eabc4f62fb2041c4b8c6586f034327ded12d92f5e7acdf04a431483f77fc102"} Nov 03 19:15:01.851750 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.851720 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerStarted","Data":"ed14c1c5f4b787305ecfcd2cc7be99943a2f0a94a55b4a834ba5dd8971187b26"} Nov 03 19:15:01.851874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.851773 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerStarted","Data":"195bb2d31fa11938b43d15dde24c202b590dfe3c8fb997b5653ef36a56a60cb6"} Nov 03 19:15:01.853519 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.853491 5988 generic.go:356] "Generic (PLEG): container finished" podID="060fc8f2-76ef-4b86-82bc-0684bc7116aa" containerID="acf8a5460069dc349bb541f813d03a485d7df96c028f411782574bbe81a9b325" exitCode=0 Nov 03 19:15:01.853622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.853574 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" event={"ID":"060fc8f2-76ef-4b86-82bc-0684bc7116aa","Type":"ContainerDied","Data":"acf8a5460069dc349bb541f813d03a485d7df96c028f411782574bbe81a9b325"} Nov 03 19:15:01.855386 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.855362 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerStarted","Data":"f0735c572af5caf6076fd3230d0d6eb6c2a171df6325d93cf490023f77c9aeca"} Nov 03 19:15:01.855499 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.855400 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerStarted","Data":"b68eb2fb276cb8b8126878e3fbbc0f299a2eadd092e71fad1d7b663a85d65ecf"} Nov 03 19:15:01.856607 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.856584 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" event={"ID":"3eedf97f-8b5d-4c39-b931-76086a20b7fa","Type":"ContainerStarted","Data":"3d771c79ee7712002794ceff270183c98c809921dc2f1cec2a9bfa92838ca839"} Nov 03 19:15:01.857968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.857946 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerStarted","Data":"a0db8291d0b61cdbed54048d26d1084a514aedf7cb09bbfebb8a7d48fa89f7f3"} Nov 03 19:15:01.858045 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.857989 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerStarted","Data":"7c2129ac38aec0777a94327671305d20c9bbb387ee688db3e047918dade83e6a"} Nov 03 19:15:01.873723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.869636 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerStarted","Data":"c4c1bdb9ed59824567385db2c6cb87206e038c9ae27677282b1d175737ce6e35"} Nov 03 19:15:01.873723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.869808 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerStarted","Data":"31a2948ce06ce3a26fe4c010e6f17c61df0f8725b06968a005402c8a04662421"} Nov 03 19:15:01.873723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.871756 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" event={"ID":"81f32a5c-5c74-4495-8d14-3bfecfe81596","Type":"ContainerStarted","Data":"57c17080fca5745f87a00130e730c249af394325b55fb98e828b76e271847bab"} Nov 03 19:15:01.873723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.873274 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" event={"ID":"31e6242f-3785-4ac5-818c-20f85285bc49","Type":"ContainerStarted","Data":"3b4a5d95a11d428b674f9bca4183f7d6c8cf6b15599832f2e2782fa3500d4fe9"} Nov 03 19:15:01.966927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:01.963058 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" podStartSLOduration=1.963038666 podStartE2EDuration="1.963038666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:01.959190377 +0000 UTC m=+2176.840606479" watchObservedRunningTime="2026-11-03 19:15:01.963038666 +0000 UTC m=+2176.844454773" Nov 03 19:15:02.070563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.070505 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" podStartSLOduration=2.070490346 podStartE2EDuration="2.070490346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.067693674 +0000 UTC m=+2176.949109983" watchObservedRunningTime="2026-11-03 19:15:02.070490346 +0000 UTC m=+2176.951906447" Nov 03 19:15:02.179368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.179311 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" podStartSLOduration=2.179296151 podStartE2EDuration="2.179296151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.175592207 +0000 UTC m=+2177.057008310" watchObservedRunningTime="2026-11-03 19:15:02.179296151 +0000 UTC m=+2177.060712252" Nov 03 19:15:02.288943 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.288898 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" podStartSLOduration=2.288885402 podStartE2EDuration="2.288885402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.285803825 +0000 UTC m=+2177.167219926" watchObservedRunningTime="2026-11-03 19:15:02.288885402 +0000 UTC m=+2177.170301504" Nov 03 19:15:02.394856 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.394808 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" podStartSLOduration=2.394792672 podStartE2EDuration="2.394792672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.393435233 +0000 UTC m=+2177.274851339" watchObservedRunningTime="2026-11-03 19:15:02.394792672 +0000 UTC m=+2177.276208773" Nov 03 19:15:02.577285 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.577257 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:15:02.673092 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673063 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-envoy\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673111 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istiod-ca-cert\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673139 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-certs\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673196 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-socket\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673306 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-podinfo\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673346 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-token\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-token\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673425 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-kube-api-access-h6nbz\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-kube-api-access-h6nbz\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673521 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673500 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-data\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673487 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-certs" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:02.673576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673564 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-credential-socket\") pod \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\" (UID: \"f929aba5-31a4-4bd0-a2bd-afc3de03159a\") " Nov 03 19:15:02.673712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673584 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-socket" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:02.673712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673602 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istiod-ca-cert" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:02.674002 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673843 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-credential-socket" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:02.674002 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.673957 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-data" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:02.674749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.674718 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.674749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.674746 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.674870 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.674763 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.674870 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.674780 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.674870 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.674796 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.675796 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.675769 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-envoy" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:02.676055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.676033 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-token" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:02.676138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.676069 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-kube-api-access-h6nbz" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "kube-api-access-h6nbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:02.676138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.676103 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-podinfo" pod "f929aba5-31a4-4bd0-a2bd-afc3de03159a" (UID: "f929aba5-31a4-4bd0-a2bd-afc3de03159a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:15:02.775807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.775728 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.775807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.775755 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.775807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.775765 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.775807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.775774 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6nbz\" (UniqueName: \"kubernetes.io/projected/f929aba5-31a4-4bd0-a2bd-afc3de03159a-kube-api-access-h6nbz\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:02.881998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.881965 5988 generic.go:356] "Generic (PLEG): container finished" podID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerID="b5d447a676aa55e7b541beba95e36af3df128affe5f1770b1125de82124d0b93" exitCode=60 Nov 03 19:15:02.882430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.882054 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" event={"ID":"3eedf97f-8b5d-4c39-b931-76086a20b7fa","Type":"ContainerDied","Data":"b5d447a676aa55e7b541beba95e36af3df128affe5f1770b1125de82124d0b93"} Nov 03 19:15:02.882430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.882298 5988 scope.go:122] "RemoveContainer" containerID="b5d447a676aa55e7b541beba95e36af3df128affe5f1770b1125de82124d0b93" Nov 03 19:15:02.883752 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.883729 5988 generic.go:356] "Generic (PLEG): container finished" podID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerID="a0db8291d0b61cdbed54048d26d1084a514aedf7cb09bbfebb8a7d48fa89f7f3" exitCode=60 Nov 03 19:15:02.883853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.883813 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerDied","Data":"a0db8291d0b61cdbed54048d26d1084a514aedf7cb09bbfebb8a7d48fa89f7f3"} Nov 03 19:15:02.884259 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.884071 5988 scope.go:122] "RemoveContainer" containerID="a0db8291d0b61cdbed54048d26d1084a514aedf7cb09bbfebb8a7d48fa89f7f3" Nov 03 19:15:02.885728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.885663 5988 generic.go:356] "Generic (PLEG): container finished" podID="453e6d0d-1386-4605-90e8-428474307891" containerID="c4c1bdb9ed59824567385db2c6cb87206e038c9ae27677282b1d175737ce6e35" exitCode=60 Nov 03 19:15:02.885728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.885712 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerDied","Data":"c4c1bdb9ed59824567385db2c6cb87206e038c9ae27677282b1d175737ce6e35"} Nov 03 19:15:02.885989 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.885972 5988 scope.go:122] "RemoveContainer" containerID="c4c1bdb9ed59824567385db2c6cb87206e038c9ae27677282b1d175737ce6e35" Nov 03 19:15:02.887624 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.887596 5988 generic.go:356] "Generic (PLEG): container finished" podID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerID="6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e" exitCode=0 Nov 03 19:15:02.887724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.887638 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" Nov 03 19:15:02.887798 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.887677 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" event={"ID":"f929aba5-31a4-4bd0-a2bd-afc3de03159a","Type":"ContainerDied","Data":"6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e"} Nov 03 19:15:02.887858 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.887805 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj" event={"ID":"f929aba5-31a4-4bd0-a2bd-afc3de03159a","Type":"ContainerDied","Data":"8a70f372a11ce29f5aa68841543d0b693145800252ac3a5d2584c263b53d1aed"} Nov 03 19:15:02.887858 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.887839 5988 scope.go:122] "RemoveContainer" containerID="6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e" Nov 03 19:15:02.889514 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.889492 5988 generic.go:356] "Generic (PLEG): container finished" podID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerID="417fe7e8d5116b34e40aed452ebdfc7d5b1ba15decfc21866bf41b8298aeada6" exitCode=60 Nov 03 19:15:02.889623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.889568 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" event={"ID":"81f32a5c-5c74-4495-8d14-3bfecfe81596","Type":"ContainerDied","Data":"417fe7e8d5116b34e40aed452ebdfc7d5b1ba15decfc21866bf41b8298aeada6"} Nov 03 19:15:02.890161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.889853 5988 scope.go:122] "RemoveContainer" containerID="417fe7e8d5116b34e40aed452ebdfc7d5b1ba15decfc21866bf41b8298aeada6" Nov 03 19:15:02.891462 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.891443 5988 generic.go:356] "Generic (PLEG): container finished" podID="31e6242f-3785-4ac5-818c-20f85285bc49" containerID="4cf8e326281e96682f6935f7f28bc176a8b6a8d9326c3f039f9195bfab410701" exitCode=60 Nov 03 19:15:02.891575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.891513 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" event={"ID":"31e6242f-3785-4ac5-818c-20f85285bc49","Type":"ContainerDied","Data":"4cf8e326281e96682f6935f7f28bc176a8b6a8d9326c3f039f9195bfab410701"} Nov 03 19:15:02.891792 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.891773 5988 scope.go:122] "RemoveContainer" containerID="4cf8e326281e96682f6935f7f28bc176a8b6a8d9326c3f039f9195bfab410701" Nov 03 19:15:02.893838 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.893647 5988 generic.go:356] "Generic (PLEG): container finished" podID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerID="8ebabb193c5b41626ff07ff0b87ac674826da8d4c8ce36d229cad90e7c7f11c3" exitCode=60 Nov 03 19:15:02.893838 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.893775 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerDied","Data":"8ebabb193c5b41626ff07ff0b87ac674826da8d4c8ce36d229cad90e7c7f11c3"} Nov 03 19:15:02.894468 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.894445 5988 scope.go:122] "RemoveContainer" containerID="8ebabb193c5b41626ff07ff0b87ac674826da8d4c8ce36d229cad90e7c7f11c3" Nov 03 19:15:02.895844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.895821 5988 generic.go:356] "Generic (PLEG): container finished" podID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerID="ed14c1c5f4b787305ecfcd2cc7be99943a2f0a94a55b4a834ba5dd8971187b26" exitCode=60 Nov 03 19:15:02.895950 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.895903 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerDied","Data":"ed14c1c5f4b787305ecfcd2cc7be99943a2f0a94a55b4a834ba5dd8971187b26"} Nov 03 19:15:02.896089 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.896073 5988 scope.go:122] "RemoveContainer" containerID="ed14c1c5f4b787305ecfcd2cc7be99943a2f0a94a55b4a834ba5dd8971187b26" Nov 03 19:15:02.897599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.897577 5988 generic.go:356] "Generic (PLEG): container finished" podID="344a6ecd-90de-4398-abff-9e246298d785" containerID="f0735c572af5caf6076fd3230d0d6eb6c2a171df6325d93cf490023f77c9aeca" exitCode=60 Nov 03 19:15:02.897741 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.897655 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerDied","Data":"f0735c572af5caf6076fd3230d0d6eb6c2a171df6325d93cf490023f77c9aeca"} Nov 03 19:15:02.897828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:02.897810 5988 scope.go:122] "RemoveContainer" containerID="f0735c572af5caf6076fd3230d0d6eb6c2a171df6325d93cf490023f77c9aeca" Nov 03 19:15:03.001576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.001545 5988 scope.go:122] "RemoveContainer" containerID="cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8" Nov 03 19:15:03.022279 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.022256 5988 scope.go:122] "RemoveContainer" containerID="6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e" Nov 03 19:15:03.023061 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:03.022725 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e\": container with ID starting with 6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e not found: ID does not exist" containerID="6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e" Nov 03 19:15:03.023061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.022768 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e"} err="failed to get container status \"6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e\": rpc error: code = NotFound desc = could not find container \"6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e\": container with ID starting with 6f41919594b27cbb53d2bf662bc730e0e799cb2f99d2bd01317981fb0f040e2e not found: ID does not exist" Nov 03 19:15:03.023061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.022794 5988 scope.go:122] "RemoveContainer" containerID="cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8" Nov 03 19:15:03.023317 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:03.023108 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8\": container with ID starting with cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8 not found: ID does not exist" containerID="cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8" Nov 03 19:15:03.023317 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.023140 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8"} err="failed to get container status \"cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8\": rpc error: code = NotFound desc = could not find container \"cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8\": container with ID starting with cbf9f47700b146f9b24dae4f28687b41b0e20d1abf8b29407bbd9000e0867cb8 not found: ID does not exist" Nov 03 19:15:03.172050 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.172021 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:15:03.186836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.186806 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-xks-ca\") pod \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " Nov 03 19:15:03.187019 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.186852 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-openshift-ca\") pod \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " Nov 03 19:15:03.187019 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.186895 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060fc8f2-76ef-4b86-82bc-0684bc7116aa-kube-api-access-pshqk\" (UniqueName: \"kubernetes.io/projected/060fc8f2-76ef-4b86-82bc-0684bc7116aa-kube-api-access-pshqk\") pod \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\" (UID: \"060fc8f2-76ef-4b86-82bc-0684bc7116aa\") " Nov 03 19:15:03.187329 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.187298 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-openshift-ca" pod "060fc8f2-76ef-4b86-82bc-0684bc7116aa" (UID: "060fc8f2-76ef-4b86-82bc-0684bc7116aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:03.187665 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.187641 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:03.189204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.189181 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060fc8f2-76ef-4b86-82bc-0684bc7116aa-kube-api-access-pshqk" pod "060fc8f2-76ef-4b86-82bc-0684bc7116aa" (UID: "060fc8f2-76ef-4b86-82bc-0684bc7116aa"). InnerVolumeSpecName "kube-api-access-pshqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:03.189285 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.189201 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-xks-ca" pod "060fc8f2-76ef-4b86-82bc-0684bc7116aa" (UID: "060fc8f2-76ef-4b86-82bc-0684bc7116aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:03.288980 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.288951 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060fc8f2-76ef-4b86-82bc-0684bc7116aa-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:03.288980 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.288981 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pshqk\" (UniqueName: \"kubernetes.io/projected/060fc8f2-76ef-4b86-82bc-0684bc7116aa-kube-api-access-pshqk\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:03.730882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.730842 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp"] Nov 03 19:15:03.754130 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.753989 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn"] Nov 03 19:15:03.904184 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.904145 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:03.908130 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.907423 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" event={"ID":"81f32a5c-5c74-4495-8d14-3bfecfe81596","Type":"ContainerStarted","Data":"77a3021ac76e83874daa02e410765e28528ab4dd9a01f0df3ca1a0c4f86b2393"} Nov 03 19:15:03.909640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.909612 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" event={"ID":"31e6242f-3785-4ac5-818c-20f85285bc49","Type":"ContainerStarted","Data":"3a185dde6436772a6103307f98ee443ec091fd147772209de4e0b6b2fc6213ea"} Nov 03 19:15:03.912004 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.911968 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerStarted","Data":"a3ec8aa3b7e288d1a7bfda547c6855734805775f5f3ff6bac002fc7a48cdc725"} Nov 03 19:15:03.914196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.914165 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerStarted","Data":"2c475c5709e1b0c45b52ee8b01a4d737d15a8ad40775f380ad2c6c406261b266"} Nov 03 19:15:03.915841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.915816 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" event={"ID":"060fc8f2-76ef-4b86-82bc-0684bc7116aa","Type":"ContainerDied","Data":"1ae06ee589c2c3e1707c07583fef2cfd7e3cfd4d54cb78764d6466e3e7fedf2a"} Nov 03 19:15:03.915959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.915862 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ae06ee589c2c3e1707c07583fef2cfd7e3cfd4d54cb78764d6466e3e7fedf2a" Nov 03 19:15:03.915959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.915819 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5" Nov 03 19:15:03.917974 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.917948 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerStarted","Data":"c41c222eba5328aa27e70d13407a0e955a345ffab0eaa3e7f2275b7a41bb1fc2"} Nov 03 19:15:03.920047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.920023 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" event={"ID":"3eedf97f-8b5d-4c39-b931-76086a20b7fa","Type":"ContainerStarted","Data":"9c901459b94e8b3588a677c1ec108f2acf17231310b95a302e372f2469e53e77"} Nov 03 19:15:03.922435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.922407 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerStarted","Data":"b4c4436128a81a70a22fff6761c366f3eba14c2e5958b032b8782f7f9e404333"} Nov 03 19:15:03.924487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.924461 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerStarted","Data":"ebb463e1761f629c323eed43f32e1f2f0f57035e045efb69df7c8c06bf1da33d"} Nov 03 19:15:03.965860 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.965827 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj"] Nov 03 19:15:03.972796 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:03.972737 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4320e5-openshift-default-656696d5c-2ltcj"] Nov 03 19:15:04.567277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.567222 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" podStartSLOduration=4.5672096920000005 podStartE2EDuration="4.567209692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:04.564630734 +0000 UTC m=+2179.446046836" watchObservedRunningTime="2026-11-03 19:15:04.567209692 +0000 UTC m=+2179.448625793" Nov 03 19:15:04.676074 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.676017 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" podStartSLOduration=4.676002224 podStartE2EDuration="4.676002224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:04.674676508 +0000 UTC m=+2179.556092609" watchObservedRunningTime="2026-11-03 19:15:04.676002224 +0000 UTC m=+2179.557418365" Nov 03 19:15:04.936313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.936218 5988 generic.go:356] "Generic (PLEG): container finished" podID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerID="2c475c5709e1b0c45b52ee8b01a4d737d15a8ad40775f380ad2c6c406261b266" exitCode=60 Nov 03 19:15:04.936313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.936266 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerDied","Data":"2c475c5709e1b0c45b52ee8b01a4d737d15a8ad40775f380ad2c6c406261b266"} Nov 03 19:15:04.936913 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.936352 5988 scope.go:122] "RemoveContainer" containerID="ed14c1c5f4b787305ecfcd2cc7be99943a2f0a94a55b4a834ba5dd8971187b26" Nov 03 19:15:04.936913 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.936757 5988 scope.go:122] "RemoveContainer" containerID="2c475c5709e1b0c45b52ee8b01a4d737d15a8ad40775f380ad2c6c406261b266" Nov 03 19:15:04.937171 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.937147 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh_odh-ai-gateway-infra(5cffe7d7-51e0-49f6-9acf-2cef2889878f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" Nov 03 19:15:04.940364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.940340 5988 generic.go:356] "Generic (PLEG): container finished" podID="344a6ecd-90de-4398-abff-9e246298d785" containerID="c41c222eba5328aa27e70d13407a0e955a345ffab0eaa3e7f2275b7a41bb1fc2" exitCode=60 Nov 03 19:15:04.940487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.940430 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerDied","Data":"c41c222eba5328aa27e70d13407a0e955a345ffab0eaa3e7f2275b7a41bb1fc2"} Nov 03 19:15:04.940769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.940752 5988 scope.go:122] "RemoveContainer" containerID="c41c222eba5328aa27e70d13407a0e955a345ffab0eaa3e7f2275b7a41bb1fc2" Nov 03 19:15:04.941104 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.941087 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm_odh-ai-gateway-infra(344a6ecd-90de-4398-abff-9e246298d785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" podUID="344a6ecd-90de-4398-abff-9e246298d785" Nov 03 19:15:04.942869 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.942850 5988 generic.go:356] "Generic (PLEG): container finished" podID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerID="9c901459b94e8b3588a677c1ec108f2acf17231310b95a302e372f2469e53e77" exitCode=60 Nov 03 19:15:04.942966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.942930 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" event={"ID":"3eedf97f-8b5d-4c39-b931-76086a20b7fa","Type":"ContainerDied","Data":"9c901459b94e8b3588a677c1ec108f2acf17231310b95a302e372f2469e53e77"} Nov 03 19:15:04.943171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.943157 5988 scope.go:122] "RemoveContainer" containerID="9c901459b94e8b3588a677c1ec108f2acf17231310b95a302e372f2469e53e77" Nov 03 19:15:04.943475 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.943447 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv_odh-ai-gateway-infra(3eedf97f-8b5d-4c39-b931-76086a20b7fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" Nov 03 19:15:04.945209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.945176 5988 generic.go:356] "Generic (PLEG): container finished" podID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerID="b4c4436128a81a70a22fff6761c366f3eba14c2e5958b032b8782f7f9e404333" exitCode=60 Nov 03 19:15:04.945313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.945221 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerDied","Data":"b4c4436128a81a70a22fff6761c366f3eba14c2e5958b032b8782f7f9e404333"} Nov 03 19:15:04.945577 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.945558 5988 scope.go:122] "RemoveContainer" containerID="b4c4436128a81a70a22fff6761c366f3eba14c2e5958b032b8782f7f9e404333" Nov 03 19:15:04.945905 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.945879 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk_odh-ai-gateway-infra(0664c8b7-6cd4-443a-9802-a7fecc7cd186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" Nov 03 19:15:04.947372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.947349 5988 generic.go:356] "Generic (PLEG): container finished" podID="453e6d0d-1386-4605-90e8-428474307891" containerID="ebb463e1761f629c323eed43f32e1f2f0f57035e045efb69df7c8c06bf1da33d" exitCode=60 Nov 03 19:15:04.947470 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.947424 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerDied","Data":"ebb463e1761f629c323eed43f32e1f2f0f57035e045efb69df7c8c06bf1da33d"} Nov 03 19:15:04.947648 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.947626 5988 scope.go:122] "RemoveContainer" containerID="ebb463e1761f629c323eed43f32e1f2f0f57035e045efb69df7c8c06bf1da33d" Nov 03 19:15:04.947912 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.947891 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8_odh-ai-gateway-infra(453e6d0d-1386-4605-90e8-428474307891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" podUID="453e6d0d-1386-4605-90e8-428474307891" Nov 03 19:15:04.949878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.949718 5988 generic.go:356] "Generic (PLEG): container finished" podID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerID="77a3021ac76e83874daa02e410765e28528ab4dd9a01f0df3ca1a0c4f86b2393" exitCode=60 Nov 03 19:15:04.949878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.949747 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" event={"ID":"81f32a5c-5c74-4495-8d14-3bfecfe81596","Type":"ContainerDied","Data":"77a3021ac76e83874daa02e410765e28528ab4dd9a01f0df3ca1a0c4f86b2393"} Nov 03 19:15:04.950079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.950059 5988 scope.go:122] "RemoveContainer" containerID="77a3021ac76e83874daa02e410765e28528ab4dd9a01f0df3ca1a0c4f86b2393" Nov 03 19:15:04.950416 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.950391 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9_odh-ai-gateway-infra(81f32a5c-5c74-4495-8d14-3bfecfe81596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" Nov 03 19:15:04.952082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.952057 5988 generic.go:356] "Generic (PLEG): container finished" podID="31e6242f-3785-4ac5-818c-20f85285bc49" containerID="3a185dde6436772a6103307f98ee443ec091fd147772209de4e0b6b2fc6213ea" exitCode=60 Nov 03 19:15:04.952185 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.952113 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" event={"ID":"31e6242f-3785-4ac5-818c-20f85285bc49","Type":"ContainerDied","Data":"3a185dde6436772a6103307f98ee443ec091fd147772209de4e0b6b2fc6213ea"} Nov 03 19:15:04.952383 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.952367 5988 scope.go:122] "RemoveContainer" containerID="3a185dde6436772a6103307f98ee443ec091fd147772209de4e0b6b2fc6213ea" Nov 03 19:15:04.952643 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.952623 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj_odh-ai-gateway-infra(31e6242f-3785-4ac5-818c-20f85285bc49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" Nov 03 19:15:04.952790 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.952678 5988 scope.go:122] "RemoveContainer" containerID="f0735c572af5caf6076fd3230d0d6eb6c2a171df6325d93cf490023f77c9aeca" Nov 03 19:15:04.954333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.954307 5988 generic.go:356] "Generic (PLEG): container finished" podID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerID="a3ec8aa3b7e288d1a7bfda547c6855734805775f5f3ff6bac002fc7a48cdc725" exitCode=60 Nov 03 19:15:04.954453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.954358 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerDied","Data":"a3ec8aa3b7e288d1a7bfda547c6855734805775f5f3ff6bac002fc7a48cdc725"} Nov 03 19:15:04.954600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.954585 5988 scope.go:122] "RemoveContainer" containerID="a3ec8aa3b7e288d1a7bfda547c6855734805775f5f3ff6bac002fc7a48cdc725" Nov 03 19:15:04.954820 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:04.954802 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr_odh-ai-gateway-infra(34f3e8db-5cf4-4793-8771-05c8075a48f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" Nov 03 19:15:04.967496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.967472 5988 scope.go:122] "RemoveContainer" containerID="b5d447a676aa55e7b541beba95e36af3df128affe5f1770b1125de82124d0b93" Nov 03 19:15:04.981068 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.981049 5988 scope.go:122] "RemoveContainer" containerID="a0db8291d0b61cdbed54048d26d1084a514aedf7cb09bbfebb8a7d48fa89f7f3" Nov 03 19:15:04.994633 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:04.994608 5988 scope.go:122] "RemoveContainer" containerID="c4c1bdb9ed59824567385db2c6cb87206e038c9ae27677282b1d175737ce6e35" Nov 03 19:15:05.009226 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.009198 5988 scope.go:122] "RemoveContainer" containerID="417fe7e8d5116b34e40aed452ebdfc7d5b1ba15decfc21866bf41b8298aeada6" Nov 03 19:15:05.023134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.023113 5988 scope.go:122] "RemoveContainer" containerID="4cf8e326281e96682f6935f7f28bc176a8b6a8d9326c3f039f9195bfab410701" Nov 03 19:15:05.036718 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.036678 5988 scope.go:122] "RemoveContainer" containerID="8ebabb193c5b41626ff07ff0b87ac674826da8d4c8ce36d229cad90e7c7f11c3" Nov 03 19:15:05.110137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.110079 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" podStartSLOduration=5.110063593 podStartE2EDuration="5.110063593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:05.10893795 +0000 UTC m=+2179.990354053" watchObservedRunningTime="2026-11-03 19:15:05.110063593 +0000 UTC m=+2179.991479692" Nov 03 19:15:05.277744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.277644 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" path="/var/lib/kubelet/pods/f929aba5-31a4-4bd0-a2bd-afc3de03159a/volumes" Nov 03 19:15:05.639651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.639609 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:05.963771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.963661 5988 scope.go:122] "RemoveContainer" containerID="b4c4436128a81a70a22fff6761c366f3eba14c2e5958b032b8782f7f9e404333" Nov 03 19:15:05.964235 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.964097 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk_odh-ai-gateway-infra(0664c8b7-6cd4-443a-9802-a7fecc7cd186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" Nov 03 19:15:05.965483 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.965465 5988 scope.go:122] "RemoveContainer" containerID="ebb463e1761f629c323eed43f32e1f2f0f57035e045efb69df7c8c06bf1da33d" Nov 03 19:15:05.965828 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.965801 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8_odh-ai-gateway-infra(453e6d0d-1386-4605-90e8-428474307891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" podUID="453e6d0d-1386-4605-90e8-428474307891" Nov 03 19:15:05.967090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.967070 5988 scope.go:122] "RemoveContainer" containerID="77a3021ac76e83874daa02e410765e28528ab4dd9a01f0df3ca1a0c4f86b2393" Nov 03 19:15:05.967358 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.967339 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9_odh-ai-gateway-infra(81f32a5c-5c74-4495-8d14-3bfecfe81596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" Nov 03 19:15:05.969114 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.968897 5988 scope.go:122] "RemoveContainer" containerID="3a185dde6436772a6103307f98ee443ec091fd147772209de4e0b6b2fc6213ea" Nov 03 19:15:05.969214 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.969159 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj_odh-ai-gateway-infra(31e6242f-3785-4ac5-818c-20f85285bc49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" Nov 03 19:15:05.970910 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.970891 5988 scope.go:122] "RemoveContainer" containerID="a3ec8aa3b7e288d1a7bfda547c6855734805775f5f3ff6bac002fc7a48cdc725" Nov 03 19:15:05.971139 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.971120 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr_odh-ai-gateway-infra(34f3e8db-5cf4-4793-8771-05c8075a48f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" Nov 03 19:15:05.972385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.972369 5988 scope.go:122] "RemoveContainer" containerID="2c475c5709e1b0c45b52ee8b01a4d737d15a8ad40775f380ad2c6c406261b266" Nov 03 19:15:05.972592 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.972574 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh_odh-ai-gateway-infra(5cffe7d7-51e0-49f6-9acf-2cef2889878f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" Nov 03 19:15:05.973899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.973881 5988 scope.go:122] "RemoveContainer" containerID="c41c222eba5328aa27e70d13407a0e955a345ffab0eaa3e7f2275b7a41bb1fc2" Nov 03 19:15:05.974089 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.974069 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm_odh-ai-gateway-infra(344a6ecd-90de-4398-abff-9e246298d785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" podUID="344a6ecd-90de-4398-abff-9e246298d785" Nov 03 19:15:05.975488 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:05.975471 5988 scope.go:122] "RemoveContainer" containerID="9c901459b94e8b3588a677c1ec108f2acf17231310b95a302e372f2469e53e77" Nov 03 19:15:05.975677 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:05.975660 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv_odh-ai-gateway-infra(3eedf97f-8b5d-4c39-b931-76086a20b7fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" Nov 03 19:15:09.832878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.832847 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:15:09.833255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.832924 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:15:09.949522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.949485 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb"] Nov 03 19:15:09.950216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950198 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" Nov 03 19:15:09.950265 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950217 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" Nov 03 19:15:09.950265 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950246 5988 cpu_manager.go:414] "containerMap: removing container" podUID="060fc8f2-76ef-4b86-82bc-0684bc7116aa" containerName="revoke-keys" Nov 03 19:15:09.950265 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950252 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060fc8f2-76ef-4b86-82bc-0684bc7116aa" containerName="revoke-keys" Nov 03 19:15:09.950361 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950273 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" Nov 03 19:15:09.950361 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950279 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" Nov 03 19:15:09.950424 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950394 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" Nov 03 19:15:09.950424 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.950407 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="060fc8f2-76ef-4b86-82bc-0684bc7116aa" containerName="revoke-keys" Nov 03 19:15:09.954063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:09.954039 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.020368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.020335 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-65e151-serving-cert\"" Nov 03 19:15:10.021298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.021272 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb"] Nov 03 19:15:10.084747 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.084641 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-metrics-certs\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.085399 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.084944 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6ws\" (UniqueName: \"kubernetes.io/projected/e25b294e-39c2-479d-b7eb-da2680d5a28c-kube-api-access-xx6ws\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.085399 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.085003 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25b294e-39c2-479d-b7eb-da2680d5a28c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.085399 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.085216 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.186181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.186150 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-metrics-certs\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.186381 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.186319 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6ws\" (UniqueName: \"kubernetes.io/projected/e25b294e-39c2-479d-b7eb-da2680d5a28c-kube-api-access-xx6ws\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.186381 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.186363 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25b294e-39c2-479d-b7eb-da2680d5a28c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.186505 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.186489 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.186653 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:10.186626 5988 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-serving-cert: secret "maas-api-e2e-inf-a-65e151-serving-cert" not found Nov 03 19:15:10.186767 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:10.186716 5988 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls podName:e25b294e-39c2-479d-b7eb-da2680d5a28c nodeName:}" failed. No retries permitted until 2026-11-03 19:15:10.686694847 +0000 UTC m=+2185.568110943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls") pod "maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" (UID: "e25b294e-39c2-479d-b7eb-da2680d5a28c") : secret "maas-api-e2e-inf-a-65e151-serving-cert" not found Nov 03 19:15:10.187193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.187172 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25b294e-39c2-479d-b7eb-da2680d5a28c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.188779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.188759 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-metrics-certs\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.228889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.228850 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:10.244938 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.244903 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6ws\" (UniqueName: \"kubernetes.io/projected/e25b294e-39c2-479d-b7eb-da2680d5a28c-kube-api-access-xx6ws\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.441742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.441645 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:10.654417 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.654370 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:10.699532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.699452 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.702010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.701988 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls\") pod \"maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:10.869089 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:10.869054 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:11.247537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:11.247427 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb"] Nov 03 19:15:11.248507 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:15:11.248474 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25b294e_39c2_479d_b7eb_da2680d5a28c.slice/crio-60c6fb32a700aacb3ef35dcde8d3b4c14e2cdc431a65301da8e5ec4e738f8f46 WatchSource:0}: Error finding container 60c6fb32a700aacb3ef35dcde8d3b4c14e2cdc431a65301da8e5ec4e738f8f46: Status 404 returned error can't find the container with id 60c6fb32a700aacb3ef35dcde8d3b4c14e2cdc431a65301da8e5ec4e738f8f46 Nov 03 19:15:12.024937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:12.024899 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" event={"ID":"e25b294e-39c2-479d-b7eb-da2680d5a28c","Type":"ContainerStarted","Data":"60c6fb32a700aacb3ef35dcde8d3b4c14e2cdc431a65301da8e5ec4e738f8f46"} Nov 03 19:15:13.034260 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:13.034219 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" event={"ID":"e25b294e-39c2-479d-b7eb-da2680d5a28c","Type":"ContainerStarted","Data":"22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418"} Nov 03 19:15:13.034631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:13.034300 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:13.147997 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:13.147945 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" podStartSLOduration=3.25326481 podStartE2EDuration="4.14792965s" totalImagesPullingTime="894.66484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:13.146047409 +0000 UTC m=+2188.027463512" watchObservedRunningTime="2026-11-03 19:15:13.14792965 +0000 UTC m=+2188.029345767" Nov 03 19:15:14.532163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:14.532118 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:14.732662 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:14.732620 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:15.217714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:15.217023 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:15.270517 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:15.270491 5988 scope.go:122] "RemoveContainer" containerID="ebb463e1761f629c323eed43f32e1f2f0f57035e045efb69df7c8c06bf1da33d" Nov 03 19:15:15.450706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:15.447550 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:16.061102 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.061019 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerStarted","Data":"96500c37207ca15268f3ea359eb363e0fe5a3259b3c6d5e7f30ddff975d31d2c"} Nov 03 19:15:16.139872 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.139841 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:16.266379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.266352 5988 scope.go:122] "RemoveContainer" containerID="77a3021ac76e83874daa02e410765e28528ab4dd9a01f0df3ca1a0c4f86b2393" Nov 03 19:15:16.266536 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.266515 5988 scope.go:122] "RemoveContainer" containerID="9c901459b94e8b3588a677c1ec108f2acf17231310b95a302e372f2469e53e77" Nov 03 19:15:16.266620 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.266570 5988 scope.go:122] "RemoveContainer" containerID="a3ec8aa3b7e288d1a7bfda547c6855734805775f5f3ff6bac002fc7a48cdc725" Nov 03 19:15:16.505557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.505521 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:16.829363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.829328 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd"] Nov 03 19:15:16.857372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.857346 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ecf40a-nvnwd"] Nov 03 19:15:16.885373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:16.885338 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:17.070035 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.070005 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" event={"ID":"81f32a5c-5c74-4495-8d14-3bfecfe81596","Type":"ContainerStarted","Data":"3380af4a9cdad1bcb977a4470b9c8e7dc4f760276896a49d222cd7e26561e977"} Nov 03 19:15:17.072044 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.072022 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerStarted","Data":"dc8f22b28e14d64cd0c78b3f9727e98182d905b9e19587397c60815b57d71079"} Nov 03 19:15:17.073914 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.073895 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" event={"ID":"3eedf97f-8b5d-4c39-b931-76086a20b7fa","Type":"ContainerStarted","Data":"82854b5802d6a5a480be49a40ff044d25f08713c27e139ed867da43e27e67248"} Nov 03 19:15:17.075783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.075762 5988 generic.go:356] "Generic (PLEG): container finished" podID="453e6d0d-1386-4605-90e8-428474307891" containerID="96500c37207ca15268f3ea359eb363e0fe5a3259b3c6d5e7f30ddff975d31d2c" exitCode=60 Nov 03 19:15:17.075863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.075838 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerDied","Data":"96500c37207ca15268f3ea359eb363e0fe5a3259b3c6d5e7f30ddff975d31d2c"} Nov 03 19:15:17.075904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.075891 5988 scope.go:122] "RemoveContainer" containerID="ebb463e1761f629c323eed43f32e1f2f0f57035e045efb69df7c8c06bf1da33d" Nov 03 19:15:17.076168 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.076151 5988 scope.go:122] "RemoveContainer" containerID="96500c37207ca15268f3ea359eb363e0fe5a3259b3c6d5e7f30ddff975d31d2c" Nov 03 19:15:17.076417 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:17.076396 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8_odh-ai-gateway-infra(453e6d0d-1386-4605-90e8-428474307891)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" podUID="453e6d0d-1386-4605-90e8-428474307891" Nov 03 19:15:17.182975 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.182939 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8"] Nov 03 19:15:17.274037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.274010 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae618a5-5a31-4cd9-99c1-d77d2899e8ab" path="/var/lib/kubelet/pods/8ae618a5-5a31-4cd9-99c1-d77d2899e8ab/volumes" Nov 03 19:15:17.355940 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:17.355894 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:18.075735 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.071651 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:18.100660 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.100623 5988 generic.go:356] "Generic (PLEG): container finished" podID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerID="3380af4a9cdad1bcb977a4470b9c8e7dc4f760276896a49d222cd7e26561e977" exitCode=60 Nov 03 19:15:18.100912 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.100742 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" event={"ID":"81f32a5c-5c74-4495-8d14-3bfecfe81596","Type":"ContainerDied","Data":"3380af4a9cdad1bcb977a4470b9c8e7dc4f760276896a49d222cd7e26561e977"} Nov 03 19:15:18.100912 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.100858 5988 scope.go:122] "RemoveContainer" containerID="77a3021ac76e83874daa02e410765e28528ab4dd9a01f0df3ca1a0c4f86b2393" Nov 03 19:15:18.101355 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.101337 5988 scope.go:122] "RemoveContainer" containerID="3380af4a9cdad1bcb977a4470b9c8e7dc4f760276896a49d222cd7e26561e977" Nov 03 19:15:18.101793 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:18.101766 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9_odh-ai-gateway-infra(81f32a5c-5c74-4495-8d14-3bfecfe81596)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" Nov 03 19:15:18.105970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.105941 5988 generic.go:356] "Generic (PLEG): container finished" podID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerID="dc8f22b28e14d64cd0c78b3f9727e98182d905b9e19587397c60815b57d71079" exitCode=60 Nov 03 19:15:18.106407 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.106203 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerDied","Data":"dc8f22b28e14d64cd0c78b3f9727e98182d905b9e19587397c60815b57d71079"} Nov 03 19:15:18.108153 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.108083 5988 scope.go:122] "RemoveContainer" containerID="dc8f22b28e14d64cd0c78b3f9727e98182d905b9e19587397c60815b57d71079" Nov 03 19:15:18.108745 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:18.108719 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr_odh-ai-gateway-infra(34f3e8db-5cf4-4793-8771-05c8075a48f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" Nov 03 19:15:18.109518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.109499 5988 generic.go:356] "Generic (PLEG): container finished" podID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerID="82854b5802d6a5a480be49a40ff044d25f08713c27e139ed867da43e27e67248" exitCode=60 Nov 03 19:15:18.109864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.109573 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" event={"ID":"3eedf97f-8b5d-4c39-b931-76086a20b7fa","Type":"ContainerDied","Data":"82854b5802d6a5a480be49a40ff044d25f08713c27e139ed867da43e27e67248"} Nov 03 19:15:18.110062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.110043 5988 scope.go:122] "RemoveContainer" containerID="82854b5802d6a5a480be49a40ff044d25f08713c27e139ed867da43e27e67248" Nov 03 19:15:18.110521 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:18.110437 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv_odh-ai-gateway-infra(3eedf97f-8b5d-4c39-b931-76086a20b7fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" Nov 03 19:15:18.129188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.129160 5988 scope.go:122] "RemoveContainer" containerID="a3ec8aa3b7e288d1a7bfda547c6855734805775f5f3ff6bac002fc7a48cdc725" Nov 03 19:15:18.148500 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.147196 5988 scope.go:122] "RemoveContainer" containerID="9c901459b94e8b3588a677c1ec108f2acf17231310b95a302e372f2469e53e77" Nov 03 19:15:18.218352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.218251 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9"] Nov 03 19:15:18.265630 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.265605 5988 scope.go:122] "RemoveContainer" containerID="b4c4436128a81a70a22fff6761c366f3eba14c2e5958b032b8782f7f9e404333" Nov 03 19:15:18.266395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.266370 5988 scope.go:122] "RemoveContainer" containerID="3a185dde6436772a6103307f98ee443ec091fd147772209de4e0b6b2fc6213ea" Nov 03 19:15:18.266647 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.266625 5988 scope.go:122] "RemoveContainer" containerID="c41c222eba5328aa27e70d13407a0e955a345ffab0eaa3e7f2275b7a41bb1fc2" Nov 03 19:15:18.267115 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.267096 5988 scope.go:122] "RemoveContainer" containerID="2c475c5709e1b0c45b52ee8b01a4d737d15a8ad40775f380ad2c6c406261b266" Nov 03 19:15:18.311830 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.311801 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr"] Nov 03 19:15:18.314209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.314187 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:18.343676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.343618 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:18.417305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.417271 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv"] Nov 03 19:15:18.528427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.528395 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453e6d0d-1386-4605-90e8-428474307891-kube-api-access-pkndt\" (UniqueName: \"kubernetes.io/projected/453e6d0d-1386-4605-90e8-428474307891-kube-api-access-pkndt\") pod \"453e6d0d-1386-4605-90e8-428474307891\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " Nov 03 19:15:18.528601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.528586 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453e6d0d-1386-4605-90e8-428474307891-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453e6d0d-1386-4605-90e8-428474307891-maas-api-service-ca\") pod \"453e6d0d-1386-4605-90e8-428474307891\" (UID: \"453e6d0d-1386-4605-90e8-428474307891\") " Nov 03 19:15:18.528987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.528963 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453e6d0d-1386-4605-90e8-428474307891-maas-api-service-ca" pod "453e6d0d-1386-4605-90e8-428474307891" (UID: "453e6d0d-1386-4605-90e8-428474307891"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:18.529130 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.529107 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/453e6d0d-1386-4605-90e8-428474307891-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.548921 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.548896 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453e6d0d-1386-4605-90e8-428474307891-kube-api-access-pkndt" pod "453e6d0d-1386-4605-90e8-428474307891" (UID: "453e6d0d-1386-4605-90e8-428474307891"). InnerVolumeSpecName "kube-api-access-pkndt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:18.630228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.630154 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkndt\" (UniqueName: \"kubernetes.io/projected/453e6d0d-1386-4605-90e8-428474307891-kube-api-access-pkndt\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:18.757565 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.757530 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" containerID="cri-o://074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251" gracePeriod=45 Nov 03 19:15:18.845869 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:18.776235 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" containerID="cri-o://55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe" gracePeriod=45 Nov 03 19:15:19.050056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.050021 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:15:19.082117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.082090 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:15:19.086709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.086691 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 19:15:19.119882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.119851 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" Nov 03 19:15:19.119882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.119876 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8" event={"ID":"453e6d0d-1386-4605-90e8-428474307891","Type":"ContainerDied","Data":"31a2948ce06ce3a26fe4c010e6f17c61df0f8725b06968a005402c8a04662421"} Nov 03 19:15:19.120210 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.119928 5988 scope.go:122] "RemoveContainer" containerID="96500c37207ca15268f3ea359eb363e0fe5a3259b3c6d5e7f30ddff975d31d2c" Nov 03 19:15:19.124584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.124557 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" event={"ID":"31e6242f-3785-4ac5-818c-20f85285bc49","Type":"ContainerStarted","Data":"6a67db3d8d27744e6a84e691aaf9e5e89e4ebfcbb28a648c634586c9645edd26"} Nov 03 19:15:19.131437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.131368 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerStarted","Data":"7ef9d2f8262ca782851bbee8328cf8bf3b478b4765159fe4e6464f74b86bbaae"} Nov 03 19:15:19.134876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.134848 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerStarted","Data":"226420538f5de3dfd55c69e7b8424970cea7f4583557bd6686e1fff9d78d5e50"} Nov 03 19:15:19.136904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.136667 5988 generic.go:356] "Generic (PLEG): container finished" podID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerID="55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe" exitCode=0 Nov 03 19:15:19.136904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.136794 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" event={"ID":"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7","Type":"ContainerDied","Data":"55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe"} Nov 03 19:15:19.136904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.136820 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" event={"ID":"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7","Type":"ContainerDied","Data":"7e5ca12bc3520cb52280477143ee7fb3d5aadbea99267e62b73a6c4d90f63538"} Nov 03 19:15:19.136904 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.136845 5988 scope.go:122] "RemoveContainer" containerID="55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe" Nov 03 19:15:19.138109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.137843 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn" Nov 03 19:15:19.140645 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.140621 5988 generic.go:356] "Generic (PLEG): container finished" podID="10baada4-97eb-4266-b663-ebd01cd83a23" containerID="074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251" exitCode=0 Nov 03 19:15:19.140838 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.140741 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" event={"ID":"10baada4-97eb-4266-b663-ebd01cd83a23","Type":"ContainerDied","Data":"074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251"} Nov 03 19:15:19.140838 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.140772 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" event={"ID":"10baada4-97eb-4266-b663-ebd01cd83a23","Type":"ContainerDied","Data":"05d25f01f5ab2309c69d2289c94e98af37186a712b698158a36b269daf1e253d"} Nov 03 19:15:19.140972 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.140838 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp" Nov 03 19:15:19.142800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.142778 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerStarted","Data":"2dcbdecd76fc692e7a04551ee12c2bb07144cecd16abe842cdf2ca2069b340a2"} Nov 03 19:15:19.148143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.148038 5988 scope.go:122] "RemoveContainer" containerID="55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe" Nov 03 19:15:19.148384 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:19.148359 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe\": container with ID starting with 55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe not found: ID does not exist" containerID="55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe" Nov 03 19:15:19.148464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.148396 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe"} err="failed to get container status \"55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe\": rpc error: code = NotFound desc = could not find container \"55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe\": container with ID starting with 55ad05082f8f1131de5dde43bbdd10c436cec7adcae50b7ffb8f6e580bfc3cfe not found: ID does not exist" Nov 03 19:15:19.148464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.148421 5988 scope.go:122] "RemoveContainer" containerID="074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251" Nov 03 19:15:19.176262 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.176222 5988 scope.go:122] "RemoveContainer" containerID="074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251" Nov 03 19:15:19.177234 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:19.177203 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251\": container with ID starting with 074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251 not found: ID does not exist" containerID="074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251" Nov 03 19:15:19.177348 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.177245 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251"} err="failed to get container status \"074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251\": rpc error: code = NotFound desc = could not find container \"074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251\": container with ID starting with 074d6acf054bde41251fe5a3140d4bf1235a37b7196c6d31d859e802df358251 not found: ID does not exist" Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238057 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-dshm\") pod \"10baada4-97eb-4266-b663-ebd01cd83a23\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238108 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tls-certs\") pod \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238149 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-kube-api-access-mxgvh\" (UniqueName: \"kubernetes.io/projected/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-kube-api-access-mxgvh\") pod \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238203 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10baada4-97eb-4266-b663-ebd01cd83a23-tls-certs\" (UniqueName: \"kubernetes.io/secret/10baada4-97eb-4266-b663-ebd01cd83a23-tls-certs\") pod \"10baada4-97eb-4266-b663-ebd01cd83a23\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238238 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-tmp-dir\") pod \"10baada4-97eb-4266-b663-ebd01cd83a23\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238293 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-model-cache\") pod \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238394 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-dshm\") pod \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " Nov 03 19:15:19.238502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238489 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10baada4-97eb-4266-b663-ebd01cd83a23-kube-api-access-p299p\" (UniqueName: \"kubernetes.io/projected/10baada4-97eb-4266-b663-ebd01cd83a23-kube-api-access-p299p\") pod \"10baada4-97eb-4266-b663-ebd01cd83a23\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " Nov 03 19:15:19.239107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238523 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-model-cache\") pod \"10baada4-97eb-4266-b663-ebd01cd83a23\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " Nov 03 19:15:19.239107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238574 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-home\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-home\") pod \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " Nov 03 19:15:19.239107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238658 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-home\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-home\") pod \"10baada4-97eb-4266-b663-ebd01cd83a23\" (UID: \"10baada4-97eb-4266-b663-ebd01cd83a23\") " Nov 03 19:15:19.239107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.238777 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tmp-dir\") pod \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\" (UID: \"295e4c83-5d79-4f4a-9e3f-fac2c816f4b7\") " Nov 03 19:15:19.245226 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.243379 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-dshm" pod "10baada4-97eb-4266-b663-ebd01cd83a23" (UID: "10baada4-97eb-4266-b663-ebd01cd83a23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.245763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.245392 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-home" pod "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" (UID: "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.245763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.245505 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tls-certs" pod "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" (UID: "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:19.245763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.245513 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-tmp-dir" pod "10baada4-97eb-4266-b663-ebd01cd83a23" (UID: "10baada4-97eb-4266-b663-ebd01cd83a23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.245763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.245568 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-model-cache" pod "10baada4-97eb-4266-b663-ebd01cd83a23" (UID: "10baada4-97eb-4266-b663-ebd01cd83a23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.246055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.245793 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-home" pod "10baada4-97eb-4266-b663-ebd01cd83a23" (UID: "10baada4-97eb-4266-b663-ebd01cd83a23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.250530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.246226 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-model-cache" pod "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" (UID: "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.250530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.246664 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tmp-dir" pod "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" (UID: "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.250530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.250009 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-kube-api-access-mxgvh" pod "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" (UID: "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7"). InnerVolumeSpecName "kube-api-access-mxgvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:19.250530 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.250449 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10baada4-97eb-4266-b663-ebd01cd83a23-tls-certs" pod "10baada4-97eb-4266-b663-ebd01cd83a23" (UID: "10baada4-97eb-4266-b663-ebd01cd83a23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:15:19.251485 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.251440 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-dshm" pod "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" (UID: "295e4c83-5d79-4f4a-9e3f-fac2c816f4b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:15:19.254617 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.254578 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10baada4-97eb-4266-b663-ebd01cd83a23-kube-api-access-p299p" pod "10baada4-97eb-4266-b663-ebd01cd83a23" (UID: "10baada4-97eb-4266-b663-ebd01cd83a23"). InnerVolumeSpecName "kube-api-access-p299p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407856 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407895 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407908 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407922 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxgvh\" (UniqueName: \"kubernetes.io/projected/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-kube-api-access-mxgvh\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407938 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10baada4-97eb-4266-b663-ebd01cd83a23-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407951 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407964 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407977 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.407989 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p299p\" (UniqueName: \"kubernetes.io/projected/10baada4-97eb-4266-b663-ebd01cd83a23-kube-api-access-p299p\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.408003 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.408016 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.408636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.408029 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10baada4-97eb-4266-b663-ebd01cd83a23-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.442845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.442754 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8"] Nov 03 19:15:19.483855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.483816 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2e052a-29895555-kt8h8"] Nov 03 19:15:19.626916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.626886 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:19.629419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.629294 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:19.637983 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.637957 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:19.725886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.725801 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f3e8db-5cf4-4793-8771-05c8075a48f6-kube-api-access-xvms9\" (UniqueName: \"kubernetes.io/projected/34f3e8db-5cf4-4793-8771-05c8075a48f6-kube-api-access-xvms9\") pod \"34f3e8db-5cf4-4793-8771-05c8075a48f6\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " Nov 03 19:15:19.725886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.725841 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f32a5c-5c74-4495-8d14-3bfecfe81596-kube-api-access-l98bq\" (UniqueName: \"kubernetes.io/projected/81f32a5c-5c74-4495-8d14-3bfecfe81596-kube-api-access-l98bq\") pod \"81f32a5c-5c74-4495-8d14-3bfecfe81596\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " Nov 03 19:15:19.725886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.725885 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eedf97f-8b5d-4c39-b931-76086a20b7fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eedf97f-8b5d-4c39-b931-76086a20b7fa-maas-api-service-ca\") pod \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " Nov 03 19:15:19.726171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.725966 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eedf97f-8b5d-4c39-b931-76086a20b7fa-kube-api-access-87f2l\" (UniqueName: \"kubernetes.io/projected/3eedf97f-8b5d-4c39-b931-76086a20b7fa-kube-api-access-87f2l\") pod \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\" (UID: \"3eedf97f-8b5d-4c39-b931-76086a20b7fa\") " Nov 03 19:15:19.726226 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726168 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f32a5c-5c74-4495-8d14-3bfecfe81596-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f32a5c-5c74-4495-8d14-3bfecfe81596-maas-api-service-ca\") pod \"81f32a5c-5c74-4495-8d14-3bfecfe81596\" (UID: \"81f32a5c-5c74-4495-8d14-3bfecfe81596\") " Nov 03 19:15:19.726226 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726205 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f3e8db-5cf4-4793-8771-05c8075a48f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f3e8db-5cf4-4793-8771-05c8075a48f6-maas-api-service-ca\") pod \"34f3e8db-5cf4-4793-8771-05c8075a48f6\" (UID: \"34f3e8db-5cf4-4793-8771-05c8075a48f6\") " Nov 03 19:15:19.726367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726323 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eedf97f-8b5d-4c39-b931-76086a20b7fa-maas-api-service-ca" pod "3eedf97f-8b5d-4c39-b931-76086a20b7fa" (UID: "3eedf97f-8b5d-4c39-b931-76086a20b7fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:19.726592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726561 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f32a5c-5c74-4495-8d14-3bfecfe81596-maas-api-service-ca" pod "81f32a5c-5c74-4495-8d14-3bfecfe81596" (UID: "81f32a5c-5c74-4495-8d14-3bfecfe81596"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:19.726592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726586 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f3e8db-5cf4-4793-8771-05c8075a48f6-maas-api-service-ca" pod "34f3e8db-5cf4-4793-8771-05c8075a48f6" (UID: "34f3e8db-5cf4-4793-8771-05c8075a48f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:19.726876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726856 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f32a5c-5c74-4495-8d14-3bfecfe81596-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.726876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726876 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f3e8db-5cf4-4793-8771-05c8075a48f6-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.727010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.726886 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eedf97f-8b5d-4c39-b931-76086a20b7fa-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.728121 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.728094 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f3e8db-5cf4-4793-8771-05c8075a48f6-kube-api-access-xvms9" pod "34f3e8db-5cf4-4793-8771-05c8075a48f6" (UID: "34f3e8db-5cf4-4793-8771-05c8075a48f6"). InnerVolumeSpecName "kube-api-access-xvms9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:19.728288 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.728268 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eedf97f-8b5d-4c39-b931-76086a20b7fa-kube-api-access-87f2l" pod "3eedf97f-8b5d-4c39-b931-76086a20b7fa" (UID: "3eedf97f-8b5d-4c39-b931-76086a20b7fa"). InnerVolumeSpecName "kube-api-access-87f2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:19.728359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.728303 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f32a5c-5c74-4495-8d14-3bfecfe81596-kube-api-access-l98bq" pod "81f32a5c-5c74-4495-8d14-3bfecfe81596" (UID: "81f32a5c-5c74-4495-8d14-3bfecfe81596"). InnerVolumeSpecName "kube-api-access-l98bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:19.828611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.828573 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvms9\" (UniqueName: \"kubernetes.io/projected/34f3e8db-5cf4-4793-8771-05c8075a48f6-kube-api-access-xvms9\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.828611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.828606 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l98bq\" (UniqueName: \"kubernetes.io/projected/81f32a5c-5c74-4495-8d14-3bfecfe81596-kube-api-access-l98bq\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:19.828611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:19.828617 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87f2l\" (UniqueName: \"kubernetes.io/projected/3eedf97f-8b5d-4c39-b931-76086a20b7fa-kube-api-access-87f2l\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:20.154371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.154339 5988 generic.go:356] "Generic (PLEG): container finished" podID="344a6ecd-90de-4398-abff-9e246298d785" containerID="226420538f5de3dfd55c69e7b8424970cea7f4583557bd6686e1fff9d78d5e50" exitCode=60 Nov 03 19:15:20.154863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.154412 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerDied","Data":"226420538f5de3dfd55c69e7b8424970cea7f4583557bd6686e1fff9d78d5e50"} Nov 03 19:15:20.154863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.154505 5988 scope.go:122] "RemoveContainer" containerID="c41c222eba5328aa27e70d13407a0e955a345ffab0eaa3e7f2275b7a41bb1fc2" Nov 03 19:15:20.155036 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.155015 5988 scope.go:122] "RemoveContainer" containerID="226420538f5de3dfd55c69e7b8424970cea7f4583557bd6686e1fff9d78d5e50" Nov 03 19:15:20.155447 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:20.155416 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm_odh-ai-gateway-infra(344a6ecd-90de-4398-abff-9e246298d785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" podUID="344a6ecd-90de-4398-abff-9e246298d785" Nov 03 19:15:20.158261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.158238 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" event={"ID":"3eedf97f-8b5d-4c39-b931-76086a20b7fa","Type":"ContainerDied","Data":"3d771c79ee7712002794ceff270183c98c809921dc2f1cec2a9bfa92838ca839"} Nov 03 19:15:20.158261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.158251 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv" Nov 03 19:15:20.162164 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.162142 5988 generic.go:356] "Generic (PLEG): container finished" podID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerID="2dcbdecd76fc692e7a04551ee12c2bb07144cecd16abe842cdf2ca2069b340a2" exitCode=60 Nov 03 19:15:20.162273 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.162220 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerDied","Data":"2dcbdecd76fc692e7a04551ee12c2bb07144cecd16abe842cdf2ca2069b340a2"} Nov 03 19:15:20.162673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.162655 5988 scope.go:122] "RemoveContainer" containerID="2dcbdecd76fc692e7a04551ee12c2bb07144cecd16abe842cdf2ca2069b340a2" Nov 03 19:15:20.163149 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:20.163116 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk_odh-ai-gateway-infra(0664c8b7-6cd4-443a-9802-a7fecc7cd186)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" Nov 03 19:15:20.165517 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.165497 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" event={"ID":"81f32a5c-5c74-4495-8d14-3bfecfe81596","Type":"ContainerDied","Data":"57c17080fca5745f87a00130e730c249af394325b55fb98e828b76e271847bab"} Nov 03 19:15:20.165517 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.165521 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9" Nov 03 19:15:20.167983 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.167961 5988 generic.go:356] "Generic (PLEG): container finished" podID="31e6242f-3785-4ac5-818c-20f85285bc49" containerID="6a67db3d8d27744e6a84e691aaf9e5e89e4ebfcbb28a648c634586c9645edd26" exitCode=60 Nov 03 19:15:20.168079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.168038 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" event={"ID":"31e6242f-3785-4ac5-818c-20f85285bc49","Type":"ContainerDied","Data":"6a67db3d8d27744e6a84e691aaf9e5e89e4ebfcbb28a648c634586c9645edd26"} Nov 03 19:15:20.168460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.168441 5988 scope.go:122] "RemoveContainer" containerID="6a67db3d8d27744e6a84e691aaf9e5e89e4ebfcbb28a648c634586c9645edd26" Nov 03 19:15:20.168839 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:20.168816 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj_odh-ai-gateway-infra(31e6242f-3785-4ac5-818c-20f85285bc49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" Nov 03 19:15:20.169622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.169519 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" Nov 03 19:15:20.169622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.169531 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr" event={"ID":"34f3e8db-5cf4-4793-8771-05c8075a48f6","Type":"ContainerDied","Data":"8eabc4f62fb2041c4b8c6586f034327ded12d92f5e7acdf04a431483f77fc102"} Nov 03 19:15:20.171871 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.171852 5988 scope.go:122] "RemoveContainer" containerID="82854b5802d6a5a480be49a40ff044d25f08713c27e139ed867da43e27e67248" Nov 03 19:15:20.171942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.171917 5988 generic.go:356] "Generic (PLEG): container finished" podID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerID="7ef9d2f8262ca782851bbee8328cf8bf3b478b4765159fe4e6464f74b86bbaae" exitCode=60 Nov 03 19:15:20.171985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.171960 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerDied","Data":"7ef9d2f8262ca782851bbee8328cf8bf3b478b4765159fe4e6464f74b86bbaae"} Nov 03 19:15:20.172292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.172277 5988 scope.go:122] "RemoveContainer" containerID="7ef9d2f8262ca782851bbee8328cf8bf3b478b4765159fe4e6464f74b86bbaae" Nov 03 19:15:20.172554 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:20.172532 5988 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh_odh-ai-gateway-infra(5cffe7d7-51e0-49f6-9acf-2cef2889878f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" Nov 03 19:15:20.182957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.182942 5988 scope.go:122] "RemoveContainer" containerID="b4c4436128a81a70a22fff6761c366f3eba14c2e5958b032b8782f7f9e404333" Nov 03 19:15:20.197850 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.197829 5988 scope.go:122] "RemoveContainer" containerID="3380af4a9cdad1bcb977a4470b9c8e7dc4f760276896a49d222cd7e26561e977" Nov 03 19:15:20.211449 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.211424 5988 scope.go:122] "RemoveContainer" containerID="3a185dde6436772a6103307f98ee443ec091fd147772209de4e0b6b2fc6213ea" Nov 03 19:15:20.224816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.224797 5988 scope.go:122] "RemoveContainer" containerID="dc8f22b28e14d64cd0c78b3f9727e98182d905b9e19587397c60815b57d71079" Nov 03 19:15:20.233572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.233555 5988 scope.go:122] "RemoveContainer" containerID="2c475c5709e1b0c45b52ee8b01a4d737d15a8ad40775f380ad2c6c406261b266" Nov 03 19:15:20.281721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.281694 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj"] Nov 03 19:15:20.552462 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.552384 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm"] Nov 03 19:15:20.626107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.626073 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr"] Nov 03 19:15:20.640961 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.640931 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-fba266-29895555-ckgwr"] Nov 03 19:15:20.808600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.808525 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9"] Nov 03 19:15:20.814974 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.814945 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-355e1a-29895555-hbzw9"] Nov 03 19:15:20.926844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.926814 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn"] Nov 03 19:15:20.929627 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.929605 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/simulator-w6-fba266-kserve-84764bc7cb-ww7rn"] Nov 03 19:15:20.986336 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:20.986308 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk"] Nov 03 19:15:21.145184 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.145148 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp"] Nov 03 19:15:21.148810 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.148783 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-fba266/distinct-w6-fba266-kserve-7bbbc6448-l95sp"] Nov 03 19:15:21.206898 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.206865 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh"] Nov 03 19:15:21.300319 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.300289 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" path="/var/lib/kubelet/pods/10baada4-97eb-4266-b663-ebd01cd83a23/volumes" Nov 03 19:15:21.301196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.301170 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" path="/var/lib/kubelet/pods/295e4c83-5d79-4f4a-9e3f-fac2c816f4b7/volumes" Nov 03 19:15:21.301903 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.301886 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" path="/var/lib/kubelet/pods/34f3e8db-5cf4-4793-8771-05c8075a48f6/volumes" Nov 03 19:15:21.303017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.302999 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453e6d0d-1386-4605-90e8-428474307891" path="/var/lib/kubelet/pods/453e6d0d-1386-4605-90e8-428474307891/volumes" Nov 03 19:15:21.303671 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.303653 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" path="/var/lib/kubelet/pods/81f32a5c-5c74-4495-8d14-3bfecfe81596/volumes" Nov 03 19:15:21.382865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.382840 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:21.431449 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.431420 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv"] Nov 03 19:15:21.436662 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.436639 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-e1f8f8-29895555-bhssv"] Nov 03 19:15:21.439169 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.439147 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:21.455070 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.455048 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e6242f-3785-4ac5-818c-20f85285bc49-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e6242f-3785-4ac5-818c-20f85285bc49-maas-api-service-ca\") pod \"31e6242f-3785-4ac5-818c-20f85285bc49\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " Nov 03 19:15:21.455204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.455123 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e6242f-3785-4ac5-818c-20f85285bc49-kube-api-access-tns8g\" (UniqueName: \"kubernetes.io/projected/31e6242f-3785-4ac5-818c-20f85285bc49-kube-api-access-tns8g\") pod \"31e6242f-3785-4ac5-818c-20f85285bc49\" (UID: \"31e6242f-3785-4ac5-818c-20f85285bc49\") " Nov 03 19:15:21.455522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.455494 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e6242f-3785-4ac5-818c-20f85285bc49-maas-api-service-ca" pod "31e6242f-3785-4ac5-818c-20f85285bc49" (UID: "31e6242f-3785-4ac5-818c-20f85285bc49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:21.455753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.455728 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e6242f-3785-4ac5-818c-20f85285bc49-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.457640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.457612 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e6242f-3785-4ac5-818c-20f85285bc49-kube-api-access-tns8g" pod "31e6242f-3785-4ac5-818c-20f85285bc49" (UID: "31e6242f-3785-4ac5-818c-20f85285bc49"). InnerVolumeSpecName "kube-api-access-tns8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:21.495563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.495541 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:21.498813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.498796 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:21.557416 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.557391 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344a6ecd-90de-4398-abff-9e246298d785-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/344a6ecd-90de-4398-abff-9e246298d785-maas-api-service-ca\") pod \"344a6ecd-90de-4398-abff-9e246298d785\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " Nov 03 19:15:21.557595 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.557453 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344a6ecd-90de-4398-abff-9e246298d785-kube-api-access-pmp5l\" (UniqueName: \"kubernetes.io/projected/344a6ecd-90de-4398-abff-9e246298d785-kube-api-access-pmp5l\") pod \"344a6ecd-90de-4398-abff-9e246298d785\" (UID: \"344a6ecd-90de-4398-abff-9e246298d785\") " Nov 03 19:15:21.557744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.557722 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344a6ecd-90de-4398-abff-9e246298d785-maas-api-service-ca" pod "344a6ecd-90de-4398-abff-9e246298d785" (UID: "344a6ecd-90de-4398-abff-9e246298d785"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:21.558169 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.558152 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tns8g\" (UniqueName: \"kubernetes.io/projected/31e6242f-3785-4ac5-818c-20f85285bc49-kube-api-access-tns8g\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.558228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.558174 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/344a6ecd-90de-4398-abff-9e246298d785-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.559551 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.559528 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344a6ecd-90de-4398-abff-9e246298d785-kube-api-access-pmp5l" pod "344a6ecd-90de-4398-abff-9e246298d785" (UID: "344a6ecd-90de-4398-abff-9e246298d785"). InnerVolumeSpecName "kube-api-access-pmp5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:21.659356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.659319 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cffe7d7-51e0-49f6-9acf-2cef2889878f-kube-api-access-tftm7\" (UniqueName: \"kubernetes.io/projected/5cffe7d7-51e0-49f6-9acf-2cef2889878f-kube-api-access-tftm7\") pod \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " Nov 03 19:15:21.659654 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.659587 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cffe7d7-51e0-49f6-9acf-2cef2889878f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cffe7d7-51e0-49f6-9acf-2cef2889878f-maas-api-service-ca\") pod \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\" (UID: \"5cffe7d7-51e0-49f6-9acf-2cef2889878f\") " Nov 03 19:15:21.659882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.659717 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0664c8b7-6cd4-443a-9802-a7fecc7cd186-kube-api-access-fkjr6\" (UniqueName: \"kubernetes.io/projected/0664c8b7-6cd4-443a-9802-a7fecc7cd186-kube-api-access-fkjr6\") pod \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " Nov 03 19:15:21.659882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.659795 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0664c8b7-6cd4-443a-9802-a7fecc7cd186-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0664c8b7-6cd4-443a-9802-a7fecc7cd186-maas-api-service-ca\") pod \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\" (UID: \"0664c8b7-6cd4-443a-9802-a7fecc7cd186\") " Nov 03 19:15:21.660255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.660198 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cffe7d7-51e0-49f6-9acf-2cef2889878f-maas-api-service-ca" pod "5cffe7d7-51e0-49f6-9acf-2cef2889878f" (UID: "5cffe7d7-51e0-49f6-9acf-2cef2889878f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:21.660840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.660529 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cffe7d7-51e0-49f6-9acf-2cef2889878f-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.660840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.660553 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp5l\" (UniqueName: \"kubernetes.io/projected/344a6ecd-90de-4398-abff-9e246298d785-kube-api-access-pmp5l\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.660840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.660558 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0664c8b7-6cd4-443a-9802-a7fecc7cd186-maas-api-service-ca" pod "0664c8b7-6cd4-443a-9802-a7fecc7cd186" (UID: "0664c8b7-6cd4-443a-9802-a7fecc7cd186"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:15:21.662756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.662725 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0664c8b7-6cd4-443a-9802-a7fecc7cd186-kube-api-access-fkjr6" pod "0664c8b7-6cd4-443a-9802-a7fecc7cd186" (UID: "0664c8b7-6cd4-443a-9802-a7fecc7cd186"). InnerVolumeSpecName "kube-api-access-fkjr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:21.662929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.662904 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cffe7d7-51e0-49f6-9acf-2cef2889878f-kube-api-access-tftm7" pod "5cffe7d7-51e0-49f6-9acf-2cef2889878f" (UID: "5cffe7d7-51e0-49f6-9acf-2cef2889878f"). InnerVolumeSpecName "kube-api-access-tftm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:15:21.761957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.761928 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkjr6\" (UniqueName: \"kubernetes.io/projected/0664c8b7-6cd4-443a-9802-a7fecc7cd186-kube-api-access-fkjr6\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.761957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.761956 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0664c8b7-6cd4-443a-9802-a7fecc7cd186-maas-api-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:21.762179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:21.761972 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftm7\" (UniqueName: \"kubernetes.io/projected/5cffe7d7-51e0-49f6-9acf-2cef2889878f-kube-api-access-tftm7\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:15:22.040725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.040424 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:22.202717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.202654 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" Nov 03 19:15:22.202717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.202666 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj" event={"ID":"31e6242f-3785-4ac5-818c-20f85285bc49","Type":"ContainerDied","Data":"3b4a5d95a11d428b674f9bca4183f7d6c8cf6b15599832f2e2782fa3500d4fe9"} Nov 03 19:15:22.202977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.202746 5988 scope.go:122] "RemoveContainer" containerID="6a67db3d8d27744e6a84e691aaf9e5e89e4ebfcbb28a648c634586c9645edd26" Nov 03 19:15:22.204277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.204261 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" Nov 03 19:15:22.204356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.204293 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh" event={"ID":"5cffe7d7-51e0-49f6-9acf-2cef2889878f","Type":"ContainerDied","Data":"195bb2d31fa11938b43d15dde24c202b590dfe3c8fb997b5653ef36a56a60cb6"} Nov 03 19:15:22.219065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.206902 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" event={"ID":"344a6ecd-90de-4398-abff-9e246298d785","Type":"ContainerDied","Data":"b68eb2fb276cb8b8126878e3fbbc0f299a2eadd092e71fad1d7b663a85d65ecf"} Nov 03 19:15:22.219065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.206959 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm" Nov 03 19:15:22.219065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.208996 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" Nov 03 19:15:22.219065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.209057 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk" event={"ID":"0664c8b7-6cd4-443a-9802-a7fecc7cd186","Type":"ContainerDied","Data":"7c2129ac38aec0777a94327671305d20c9bbb387ee688db3e047918dade83e6a"} Nov 03 19:15:22.219065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.217155 5988 scope.go:122] "RemoveContainer" containerID="7ef9d2f8262ca782851bbee8328cf8bf3b478b4765159fe4e6464f74b86bbaae" Nov 03 19:15:22.232165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.232147 5988 scope.go:122] "RemoveContainer" containerID="226420538f5de3dfd55c69e7b8424970cea7f4583557bd6686e1fff9d78d5e50" Nov 03 19:15:22.241821 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.241805 5988 scope.go:122] "RemoveContainer" containerID="2dcbdecd76fc692e7a04551ee12c2bb07144cecd16abe842cdf2ca2069b340a2" Nov 03 19:15:22.356916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.356887 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm"] Nov 03 19:15:22.361205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.361181 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a93912-29895555-966fm"] Nov 03 19:15:22.463408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.463383 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk"] Nov 03 19:15:22.467339 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.467319 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-69e2b9-29895555-ws4mk"] Nov 03 19:15:22.569480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.569451 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh"] Nov 03 19:15:22.573104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.573082 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9e277a-29895555-ch7qh"] Nov 03 19:15:22.677305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.677218 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj"] Nov 03 19:15:22.680103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:22.680074 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-3a7056-29895555-fn5dj"] Nov 03 19:15:23.277877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:23.277839 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" path="/var/lib/kubelet/pods/0664c8b7-6cd4-443a-9802-a7fecc7cd186/volumes" Nov 03 19:15:23.278599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:23.278573 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" path="/var/lib/kubelet/pods/31e6242f-3785-4ac5-818c-20f85285bc49/volumes" Nov 03 19:15:23.279228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:23.279207 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344a6ecd-90de-4398-abff-9e246298d785" path="/var/lib/kubelet/pods/344a6ecd-90de-4398-abff-9e246298d785/volumes" Nov 03 19:15:23.280289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:23.280267 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" path="/var/lib/kubelet/pods/3eedf97f-8b5d-4c39-b931-76086a20b7fa/volumes" Nov 03 19:15:23.280931 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:23.280912 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" path="/var/lib/kubelet/pods/5cffe7d7-51e0-49f6-9acf-2cef2889878f/volumes" Nov 03 19:15:24.823669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:24.823606 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:25.076760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:25.076497 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:25.318563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:25.316165 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:25.609444 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:25.609402 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:25.815255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:25.815196 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:25.940278 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:25.940193 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:15:27.016020 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:27.015979 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:31.994187 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:31.994134 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:36.959878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:36.959817 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:39.077184 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:39.077146 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:39.287988 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:39.287942 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:39.983767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:39.983729 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:40.187862 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:40.187826 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:41.628458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:41.628420 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:42.098313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:42.098281 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:44.486840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:44.486805 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:15:44.487240 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:44.486867 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:15:45.476229 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:45.476133 5988 generic.go:356] "Generic (PLEG): container finished" podID="83803762-e846-4658-812c-a5dc412ebb1f" containerID="0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f" exitCode=0 Nov 03 19:15:45.476229 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:45.476199 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" event={"ID":"83803762-e846-4658-812c-a5dc412ebb1f","Type":"ContainerDied","Data":"0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f"} Nov 03 19:15:45.476822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:45.476796 5988 scope.go:122] "RemoveContainer" containerID="0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f" Nov 03 19:15:45.479005 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:45.478974 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:15:45.479090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:45.479044 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:15:45.479090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:45.479077 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:15:46.484660 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:46.484625 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" event={"ID":"83803762-e846-4658-812c-a5dc412ebb1f","Type":"ContainerStarted","Data":"6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a"} Nov 03 19:15:46.525773 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:46.525747 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:15:46.526023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:46.525997 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:15:46.526074 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:46.526043 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 19:15:47.023409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:47.023374 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:47.525485 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:47.525450 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:15:47.525929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:47.525510 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 19:15:47.580349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:47.580308 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:47.794783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:47.794669 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:48.525465 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:48.525434 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:15:48.525631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:48.525491 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 19:15:49.525572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:49.525538 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:15:49.525761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:49.525587 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 19:15:50.525363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:50.525328 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:15:50.525544 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:50.525394 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 19:15:51.525232 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:51.525198 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:15:51.525616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:51.525262 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 19:15:51.967360 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:51.967321 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:52.525252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:52.525219 5988 patch_prober.go:29] interesting pod/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 19:15:52.525668 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:52.525280 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 19:15:53.530786 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:53.530756 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:15:53.531259 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:53.531239 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:15:53.532109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:53.532090 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:15:53.783222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:53.783130 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:15:55.937373 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:15:55.937325 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:15:56.079973 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:56.079944 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:15:56.630706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:15:56.627666 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:01.153918 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:01.153879 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:05.663489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:05.663431 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:10.165890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:10.165846 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:14.823980 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:14.823939 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:18.151389 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:18.151347 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:19.252570 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:19.252531 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:19.450930 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:19.450893 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:24.005269 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:24.004995 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:25.937157 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:16:25.937117 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:16:26.224867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:26.224793 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4"] Nov 03 19:16:26.225018 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:26.224996 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" containerID="cri-o://e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4" gracePeriod=30 Nov 03 19:16:27.941325 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:27.941287 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:29.038707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:29.038639 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:29.491555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:29.491521 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:29.732034 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:29.732003 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:29.950172 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:29.950137 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:30.477065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:30.477031 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:31.260899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.260869 5988 patch_prober.go:29] interesting pod/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 19:16:31.261003 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.260919 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 19:16:31.497173 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.497149 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:16:31.649778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.649748 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-podinfo\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.649922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.649818 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-kube-api-access-rj6lw\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-kube-api-access-rj6lw\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.649922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.649845 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-certs\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.649922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.649873 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-socket\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.650057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.649967 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-data\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.650057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650000 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-envoy\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.650057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650028 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-token\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-token\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.650171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650080 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istiod-ca-cert\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.650171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650154 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-credential-socket\") pod \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\" (UID: \"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce\") " Nov 03 19:16:31.650252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650169 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-certs" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:31.650421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650380 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-data" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:31.650567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650497 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-credential-socket" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:31.650567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650510 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istiod-ca-cert" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:31.650779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650754 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-socket" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:31.650881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650788 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.650881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650811 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.650881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650825 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.650881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.650842 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.652248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.652224 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-kube-api-access-rj6lw" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "kube-api-access-rj6lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:31.652425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.652404 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-podinfo" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:31.652511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.652488 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-token" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:31.652752 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.652715 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-envoy" pod "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" (UID: "e8de12aa-96f8-4e54-ad0d-6e5fa89202ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:31.752261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.752174 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.752261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.752207 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj6lw\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-kube-api-access-rj6lw\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.752261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.752222 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.752261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.752236 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.752261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.752249 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:31.794541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.794507 5988 generic.go:356] "Generic (PLEG): container finished" podID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerID="e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4" exitCode=0 Nov 03 19:16:31.794742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.794552 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" Nov 03 19:16:31.794742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.794597 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" event={"ID":"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce","Type":"ContainerDied","Data":"e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4"} Nov 03 19:16:31.794742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.794658 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4" event={"ID":"e8de12aa-96f8-4e54-ad0d-6e5fa89202ce","Type":"ContainerDied","Data":"5bacf69ea61ab5383fef1f764d7e4167127d60a34a38bdbb6ddc9f1e85e6db5b"} Nov 03 19:16:31.794742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.794697 5988 scope.go:122] "RemoveContainer" containerID="e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4" Nov 03 19:16:31.805797 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.805773 5988 scope.go:122] "RemoveContainer" containerID="e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4" Nov 03 19:16:31.806072 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:16:31.806045 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4\": container with ID starting with e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4 not found: ID does not exist" containerID="e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4" Nov 03 19:16:31.806161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.806081 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4"} err="failed to get container status \"e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4\": rpc error: code = NotFound desc = could not find container \"e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4\": container with ID starting with e609c58d904aaa44b7199299fb19bf605e77266b84267e5de9b5211e09a7b3c4 not found: ID does not exist" Nov 03 19:16:31.974286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.974256 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4"] Nov 03 19:16:31.978514 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:31.978489 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9686c3-openshift-default-85685b9d55-xk4f4"] Nov 03 19:16:33.273390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:33.273356 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" path="/var/lib/kubelet/pods/e8de12aa-96f8-4e54-ad0d-6e5fa89202ce/volumes" Nov 03 19:16:34.789675 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:34.789636 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:36.828238 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:36.828194 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:37.035317 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:37.035277 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:37.125230 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:37.125153 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62"] Nov 03 19:16:37.125375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:37.125352 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" containerID="cri-o://a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5" gracePeriod=30 Nov 03 19:16:37.267704 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:37.266918 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:38.371114 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:38.371083 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:39.012767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.012701 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:39.031130 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.031095 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc"] Nov 03 19:16:39.031788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.031724 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" podUID="f119ec89-accf-40b3-87f7-dcf59d978d0f" containerName="maas-api" containerID="cri-o://4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864" gracePeriod=30 Nov 03 19:16:39.227733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.227676 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:39.435160 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.435122 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:39.502761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.502727 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 19:16:39.522767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.522735 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls\") pod \"f119ec89-accf-40b3-87f7-dcf59d978d0f\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " Nov 03 19:16:39.522929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.522899 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-metrics-certs\") pod \"f119ec89-accf-40b3-87f7-dcf59d978d0f\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " Nov 03 19:16:39.522990 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.522974 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f119ec89-accf-40b3-87f7-dcf59d978d0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f119ec89-accf-40b3-87f7-dcf59d978d0f-openshift-service-ca\") pod \"f119ec89-accf-40b3-87f7-dcf59d978d0f\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " Nov 03 19:16:39.523096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.523073 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f119ec89-accf-40b3-87f7-dcf59d978d0f-kube-api-access-pq82d\" (UniqueName: \"kubernetes.io/projected/f119ec89-accf-40b3-87f7-dcf59d978d0f-kube-api-access-pq82d\") pod \"f119ec89-accf-40b3-87f7-dcf59d978d0f\" (UID: \"f119ec89-accf-40b3-87f7-dcf59d978d0f\") " Nov 03 19:16:39.523326 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.523306 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f119ec89-accf-40b3-87f7-dcf59d978d0f-openshift-service-ca" pod "f119ec89-accf-40b3-87f7-dcf59d978d0f" (UID: "f119ec89-accf-40b3-87f7-dcf59d978d0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:39.529811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.523639 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f119ec89-accf-40b3-87f7-dcf59d978d0f-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:39.529811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.525494 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f119ec89-accf-40b3-87f7-dcf59d978d0f-kube-api-access-pq82d" pod "f119ec89-accf-40b3-87f7-dcf59d978d0f" (UID: "f119ec89-accf-40b3-87f7-dcf59d978d0f"). InnerVolumeSpecName "kube-api-access-pq82d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:39.529811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.525501 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls" pod "f119ec89-accf-40b3-87f7-dcf59d978d0f" (UID: "f119ec89-accf-40b3-87f7-dcf59d978d0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:39.529811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.525531 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-metrics-certs" pod "f119ec89-accf-40b3-87f7-dcf59d978d0f" (UID: "f119ec89-accf-40b3-87f7-dcf59d978d0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:39.628988 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.628905 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:39.629195 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.629178 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq82d\" (UniqueName: \"kubernetes.io/projected/f119ec89-accf-40b3-87f7-dcf59d978d0f-kube-api-access-pq82d\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:39.629282 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.629271 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f119ec89-accf-40b3-87f7-dcf59d978d0f-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:39.763040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.762777 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57"] Nov 03 19:16:39.763040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.762984 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" containerID="cri-o://785f789f233fdba23bfec20b0c37a2ebac322b8d6a623b5ef6dc7bb20c8a45a1" gracePeriod=30 Nov 03 19:16:39.774933 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.774892 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr"] Nov 03 19:16:39.775116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.775092 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" containerID="cri-o://8b73a4db54a30257081cb70b2e9805ada54b61f7a4ca9af95397e3a345e4d518" gracePeriod=30 Nov 03 19:16:39.783367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.783332 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh"] Nov 03 19:16:39.783604 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.783562 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" containerID="cri-o://e8fe6c2b6bb27eea35b100ad25ae99c15a721c0a871fb7dc8e1c2e9f8f5e6d7a" gracePeriod=30 Nov 03 19:16:39.864339 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.864305 5988 generic.go:356] "Generic (PLEG): container finished" podID="f119ec89-accf-40b3-87f7-dcf59d978d0f" containerID="4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864" exitCode=0 Nov 03 19:16:39.864586 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.864357 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" Nov 03 19:16:39.864735 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.864375 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" event={"ID":"f119ec89-accf-40b3-87f7-dcf59d978d0f","Type":"ContainerDied","Data":"4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864"} Nov 03 19:16:39.864803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.864782 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc" event={"ID":"f119ec89-accf-40b3-87f7-dcf59d978d0f","Type":"ContainerDied","Data":"755849a6d3ba1d83fce41801bb27005c0606050fbdd7f543040d9eb9ecc2e0c0"} Nov 03 19:16:39.864872 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.864821 5988 scope.go:122] "RemoveContainer" containerID="4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864" Nov 03 19:16:39.881301 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.881280 5988 scope.go:122] "RemoveContainer" containerID="4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864" Nov 03 19:16:39.881606 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:16:39.881584 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864\": container with ID starting with 4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864 not found: ID does not exist" containerID="4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864" Nov 03 19:16:39.881666 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.881616 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864"} err="failed to get container status \"4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864\": rpc error: code = NotFound desc = could not find container \"4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864\": container with ID starting with 4fd66232b2a7705bdd40d10c5f6d251f331709db80d6452449c9e0252c13f864 not found: ID does not exist" Nov 03 19:16:40.000077 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:39.999920 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc"] Nov 03 19:16:40.002575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.002548 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-fba266-6697f76596-h4ncc"] Nov 03 19:16:40.446915 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.446882 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg"] Nov 03 19:16:40.447364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.447124 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" podUID="1c89b584-f9f5-4ef6-8a1f-94586db8d09b" containerName="maas-api" containerID="cri-o://4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15" gracePeriod=30 Nov 03 19:16:40.612288 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.612255 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:40.788079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.788048 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:16:40.846188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.846152 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-kube-api-access-c9dqx\" (UniqueName: \"kubernetes.io/projected/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-kube-api-access-c9dqx\") pod \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " Nov 03 19:16:40.846352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.846314 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-metrics-certs\") pod \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " Nov 03 19:16:40.846395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.846352 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-openshift-service-ca\") pod \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " Nov 03 19:16:40.846467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.846439 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls\") pod \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\" (UID: \"1c89b584-f9f5-4ef6-8a1f-94586db8d09b\") " Nov 03 19:16:40.846781 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.846753 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-openshift-service-ca" pod "1c89b584-f9f5-4ef6-8a1f-94586db8d09b" (UID: "1c89b584-f9f5-4ef6-8a1f-94586db8d09b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:40.847082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.847057 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:40.848511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.848483 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls" pod "1c89b584-f9f5-4ef6-8a1f-94586db8d09b" (UID: "1c89b584-f9f5-4ef6-8a1f-94586db8d09b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:40.848610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.848562 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-metrics-certs" pod "1c89b584-f9f5-4ef6-8a1f-94586db8d09b" (UID: "1c89b584-f9f5-4ef6-8a1f-94586db8d09b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:16:40.848805 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.848777 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-kube-api-access-c9dqx" pod "1c89b584-f9f5-4ef6-8a1f-94586db8d09b" (UID: "1c89b584-f9f5-4ef6-8a1f-94586db8d09b"). InnerVolumeSpecName "kube-api-access-c9dqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:40.875076 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.875047 5988 generic.go:356] "Generic (PLEG): container finished" podID="1c89b584-f9f5-4ef6-8a1f-94586db8d09b" containerID="4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15" exitCode=0 Nov 03 19:16:40.875212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.875087 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" Nov 03 19:16:40.875212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.875126 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" event={"ID":"1c89b584-f9f5-4ef6-8a1f-94586db8d09b","Type":"ContainerDied","Data":"4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15"} Nov 03 19:16:40.875212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.875176 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg" event={"ID":"1c89b584-f9f5-4ef6-8a1f-94586db8d09b","Type":"ContainerDied","Data":"f8fcc16a6198275480b7346d37b4a74cb5a05d56c369c235f90734953f45a0f2"} Nov 03 19:16:40.875212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.875203 5988 scope.go:122] "RemoveContainer" containerID="4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15" Nov 03 19:16:40.887199 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.887174 5988 scope.go:122] "RemoveContainer" containerID="4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15" Nov 03 19:16:40.887525 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:16:40.887500 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15\": container with ID starting with 4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15 not found: ID does not exist" containerID="4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15" Nov 03 19:16:40.887611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.887540 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15"} err="failed to get container status \"4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15\": rpc error: code = NotFound desc = could not find container \"4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15\": container with ID starting with 4fda8a5ce002e753d1d8b33684f06752fe8618401ed1532d57f36a9ac8fc8f15 not found: ID does not exist" Nov 03 19:16:40.945769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.945715 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:40.948601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.948563 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:40.948788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.948606 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9dqx\" (UniqueName: \"kubernetes.io/projected/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-kube-api-access-c9dqx\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:40.948788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:40.948626 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c89b584-f9f5-4ef6-8a1f-94586db8d09b-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:41.006430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.006346 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg"] Nov 03 19:16:41.014783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.014754 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a93912-86c79b867b-l5tzg"] Nov 03 19:16:41.246431 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.246398 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:41.283542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.283449 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c89b584-f9f5-4ef6-8a1f-94586db8d09b" path="/var/lib/kubelet/pods/1c89b584-f9f5-4ef6-8a1f-94586db8d09b/volumes" Nov 03 19:16:41.284303 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.284276 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f119ec89-accf-40b3-87f7-dcf59d978d0f" path="/var/lib/kubelet/pods/f119ec89-accf-40b3-87f7-dcf59d978d0f/volumes" Nov 03 19:16:41.454576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.454544 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:41.566815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.562745 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5"] Nov 03 19:16:41.572982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.572858 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-fba266-tppp5"] Nov 03 19:16:41.600529 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.600495 5988 patch_prober.go:29] interesting pod/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:16:41.600828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:41.600795 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:16:42.007919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.007844 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:42.203859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.203826 5988 patch_prober.go:29] interesting pod/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:16:42.204012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.203886 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:16:42.212577 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.212552 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:42.318724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.318695 5988 patch_prober.go:29] interesting pod/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:16:42.318895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.318751 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:16:42.413565 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.413540 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:16:42.415186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.415155 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492413 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-certs\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492568 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-podinfo\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492709 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-socket\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492758 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-envoy\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492807 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-token\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492838 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-data\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492886 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de86ec7-7982-4771-a8a8-c62921c86eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de86ec7-7982-4771-a8a8-c62921c86eb6-istiod-ca-cert\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.492916 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-kube-api-access-zsdtl\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-kube-api-access-zsdtl\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.493649 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-data" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.493975 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-certs" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.494127 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-socket" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.494357 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de86ec7-7982-4771-a8a8-c62921c86eb6-istiod-ca-cert" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.494446 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-credential-socket\") pod \"5de86ec7-7982-4771-a8a8-c62921c86eb6\" (UID: \"5de86ec7-7982-4771-a8a8-c62921c86eb6\") " Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.497630 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-credential-socket" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.497874 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.497978 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.497996 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5de86ec7-7982-4771-a8a8-c62921c86eb6-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.498012 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.498496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.498032 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.507728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.500400 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-token" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:42.507728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.501956 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-envoy" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:42.507728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.502730 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-podinfo" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:42.515290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.515250 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-kube-api-access-zsdtl" pod "5de86ec7-7982-4771-a8a8-c62921c86eb6" (UID: "5de86ec7-7982-4771-a8a8-c62921c86eb6"). InnerVolumeSpecName "kube-api-access-zsdtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:42.599769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.599736 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.599769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.599764 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.599769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.599773 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.600006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.599783 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdtl\" (UniqueName: \"kubernetes.io/projected/5de86ec7-7982-4771-a8a8-c62921c86eb6-kube-api-access-zsdtl\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:42.625863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.625833 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:42.840582 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.840546 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:42.901546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.901455 5988 generic.go:356] "Generic (PLEG): container finished" podID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerID="a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5" exitCode=0 Nov 03 19:16:42.903159 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.903024 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" Nov 03 19:16:42.904427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.903047 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" event={"ID":"5de86ec7-7982-4771-a8a8-c62921c86eb6","Type":"ContainerDied","Data":"a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5"} Nov 03 19:16:42.904857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.904822 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62" event={"ID":"5de86ec7-7982-4771-a8a8-c62921c86eb6","Type":"ContainerDied","Data":"8bb2d647dd14a282e827f1b572a0c1020c0577832b8c8c1332eca2ad2039d7c0"} Nov 03 19:16:42.904978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.904882 5988 scope.go:122] "RemoveContainer" containerID="a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5" Nov 03 19:16:42.919608 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.919585 5988 scope.go:122] "RemoveContainer" containerID="a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5" Nov 03 19:16:42.919926 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:16:42.919905 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5\": container with ID starting with a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5 not found: ID does not exist" containerID="a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5" Nov 03 19:16:42.920011 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:42.919940 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5"} err="failed to get container status \"a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5\": rpc error: code = NotFound desc = could not find container \"a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5\": container with ID starting with a0d381600e377bf56fad16d4240afe01f4572c5626ea9701173b7e6c61be33b5 not found: ID does not exist" Nov 03 19:16:43.061549 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:43.061511 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62"] Nov 03 19:16:43.107799 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:43.107759 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-97e020-openshift-default-777b4d5657-f7z62"] Nov 03 19:16:43.274143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:43.274067 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060fc8f2-76ef-4b86-82bc-0684bc7116aa" path="/var/lib/kubelet/pods/060fc8f2-76ef-4b86-82bc-0684bc7116aa/volumes" Nov 03 19:16:43.274604 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:43.274589 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" path="/var/lib/kubelet/pods/5de86ec7-7982-4771-a8a8-c62921c86eb6/volumes" Nov 03 19:16:43.328128 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:43.328100 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:44.173417 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.173363 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:44.402888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.402853 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:44.614827 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.614792 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:44.823490 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.823457 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:44.920851 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.920648 5988 generic.go:356] "Generic (PLEG): container finished" podID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerID="e8fe6c2b6bb27eea35b100ad25ae99c15a721c0a871fb7dc8e1c2e9f8f5e6d7a" exitCode=0 Nov 03 19:16:44.920851 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.920766 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" event={"ID":"cb291fbd-f786-4780-8fff-e07e380f99b9","Type":"ContainerDied","Data":"e8fe6c2b6bb27eea35b100ad25ae99c15a721c0a871fb7dc8e1c2e9f8f5e6d7a"} Nov 03 19:16:44.924015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.922784 5988 generic.go:356] "Generic (PLEG): container finished" podID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerID="785f789f233fdba23bfec20b0c37a2ebac322b8d6a623b5ef6dc7bb20c8a45a1" exitCode=0 Nov 03 19:16:44.924015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.922870 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" event={"ID":"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9","Type":"ContainerDied","Data":"785f789f233fdba23bfec20b0c37a2ebac322b8d6a623b5ef6dc7bb20c8a45a1"} Nov 03 19:16:44.925657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.924619 5988 generic.go:356] "Generic (PLEG): container finished" podID="b665b35c-7648-4d19-b8d3-e723680182a6" containerID="8b73a4db54a30257081cb70b2e9805ada54b61f7a4ca9af95397e3a345e4d518" exitCode=0 Nov 03 19:16:44.925657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:44.924667 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" event={"ID":"b665b35c-7648-4d19-b8d3-e723680182a6","Type":"ContainerDied","Data":"8b73a4db54a30257081cb70b2e9805ada54b61f7a4ca9af95397e3a345e4d518"} Nov 03 19:16:45.076786 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.076753 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 19:16:45.106370 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.106344 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:16:45.125664 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.125642 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:16:45.132527 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132507 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-data\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.132621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132565 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-podinfo\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.132621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132587 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-token\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-token\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.132621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132610 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b665b35c-7648-4d19-b8d3-e723680182a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b665b35c-7648-4d19-b8d3-e723680182a6-istio-podinfo\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.132800 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132649 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-kube-api-access-dgpdm\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-kube-api-access-dgpdm\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.132861 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132817 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-data" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.133252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132931 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b665b35c-7648-4d19-b8d3-e723680182a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b665b35c-7648-4d19-b8d3-e723680182a6-istiod-ca-cert\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.133252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.132999 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-socket\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.133252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133048 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-data\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.133252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133110 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-certs\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.133252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133155 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-certs\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.133252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133243 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istiod-ca-cert\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.133601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133305 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-kube-api-access-fn2jm\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-kube-api-access-fn2jm\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.133601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133356 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-socket\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.133601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133416 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-credential-socket\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.133601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133463 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-envoy\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.133601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133504 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-envoy\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.133601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133549 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-istio-token\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-istio-token\") pod \"b665b35c-7648-4d19-b8d3-e723680182a6\" (UID: \"b665b35c-7648-4d19-b8d3-e723680182a6\") " Nov 03 19:16:45.133964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133354 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-socket" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.133964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133379 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b665b35c-7648-4d19-b8d3-e723680182a6-istiod-ca-cert" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:45.133964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133626 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-socket" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.133964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133646 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-certs" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.133964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133833 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-certs" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.133964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.133908 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-credential-socket" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.134268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134237 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-credential-socket\") pod \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\" (UID: \"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9\") " Nov 03 19:16:45.134386 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134363 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istiod-ca-cert" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:45.134448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134427 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-data" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.134867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134830 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-credential-socket" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.134962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134911 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.134962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134928 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b665b35c-7648-4d19-b8d3-e723680182a6-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.134962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134944 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.134962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134958 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.135186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134973 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.135186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.134987 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.135186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.135003 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.135186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.135018 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.135186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.135032 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.135532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.135489 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-podinfo" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:45.135599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.135558 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b665b35c-7648-4d19-b8d3-e723680182a6-istio-podinfo" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:45.135985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.135957 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-kube-api-access-fn2jm" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "kube-api-access-fn2jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:45.136102 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.136003 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-token" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:45.136244 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.136225 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-envoy" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.136429 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.136411 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-kube-api-access-dgpdm" pod "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" (UID: "3afd215e-7b95-4ef7-b1d5-1ab1c58764e9"). InnerVolumeSpecName "kube-api-access-dgpdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:45.136927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.136912 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-envoy" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.137087 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.137069 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-istio-token" pod "b665b35c-7648-4d19-b8d3-e723680182a6" (UID: "b665b35c-7648-4d19-b8d3-e723680182a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:45.243753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.236590 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-token\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-token\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.243753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.236737 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-socket\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.243753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.236893 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-podinfo\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.243753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.236945 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb291fbd-f786-4780-8fff-e07e380f99b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb291fbd-f786-4780-8fff-e07e380f99b9-istiod-ca-cert\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.243753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.236992 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-certs\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.251715 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.250555 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb291fbd-f786-4780-8fff-e07e380f99b9-istiod-ca-cert" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:45.260502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.259826 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-certs" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.260980 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-data\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.261058 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-kube-api-access-njt9k\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-kube-api-access-njt9k\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.261143 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-credential-socket\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.261188 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-envoy\") pod \"cb291fbd-f786-4780-8fff-e07e380f99b9\" (UID: \"cb291fbd-f786-4780-8fff-e07e380f99b9\") " Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262246 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-socket" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262422 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb291fbd-f786-4780-8fff-e07e380f99b9-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262442 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262458 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn2jm\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-kube-api-access-fn2jm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262473 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b665b35c-7648-4d19-b8d3-e723680182a6-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262487 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262502 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b665b35c-7648-4d19-b8d3-e723680182a6-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262517 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262531 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262546 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262565 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b665b35c-7648-4d19-b8d3-e723680182a6-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262579 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgpdm\" (UniqueName: \"kubernetes.io/projected/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9-kube-api-access-dgpdm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262594 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.262840 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-data" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.267507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.264013 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-credential-socket" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.281769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.281732 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-kube-api-access-njt9k" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "kube-api-access-njt9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:45.309677 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.309588 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-podinfo" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:45.309850 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.309715 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-token" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:45.323009 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.322947 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-envoy" pod "cb291fbd-f786-4780-8fff-e07e380f99b9" (UID: "cb291fbd-f786-4780-8fff-e07e380f99b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:45.364532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.364505 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.364532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.364532 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.364532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.364547 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njt9k\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-kube-api-access-njt9k\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.364810 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.364564 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.364810 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.364577 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.364810 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.364589 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb291fbd-f786-4780-8fff-e07e380f99b9-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:45.603564 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.603524 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:16:45.932742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.932658 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" event={"ID":"cb291fbd-f786-4780-8fff-e07e380f99b9","Type":"ContainerDied","Data":"a47e1d6ea5a2603f22a26f59ee7bfec3a6a56f0b5366f382078625e27e19884b"} Nov 03 19:16:45.932742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.932708 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh" Nov 03 19:16:45.932982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.932743 5988 scope.go:122] "RemoveContainer" containerID="e8fe6c2b6bb27eea35b100ad25ae99c15a721c0a871fb7dc8e1c2e9f8f5e6d7a" Nov 03 19:16:45.934375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.934352 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" event={"ID":"3afd215e-7b95-4ef7-b1d5-1ab1c58764e9","Type":"ContainerDied","Data":"0c9669e6cffd6d2eb07f344319464d2f11a5bb87e16dd66d985246de30413093"} Nov 03 19:16:45.934375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.934365 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57" Nov 03 19:16:45.935905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.935881 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" Nov 03 19:16:45.935905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.935894 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr" event={"ID":"b665b35c-7648-4d19-b8d3-e723680182a6","Type":"ContainerDied","Data":"1576740bbc0989f3f97889c585d584b681fe0dae8131a077f4567b8343402652"} Nov 03 19:16:45.944636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.944616 5988 scope.go:122] "RemoveContainer" containerID="785f789f233fdba23bfec20b0c37a2ebac322b8d6a623b5ef6dc7bb20c8a45a1" Nov 03 19:16:45.953869 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:45.953848 5988 scope.go:122] "RemoveContainer" containerID="8b73a4db54a30257081cb70b2e9805ada54b61f7a4ca9af95397e3a345e4d518" Nov 03 19:16:46.088804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:46.088768 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh"] Nov 03 19:16:46.134850 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:46.134828 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-5kgqh"] Nov 03 19:16:46.217779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:46.205570 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57"] Nov 03 19:16:46.217779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:46.208718 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-355e1a-openshift-default-d7d9459f9-b6d57"] Nov 03 19:16:46.315500 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:46.315468 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr"] Nov 03 19:16:46.319378 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:46.319348 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-xp5mr"] Nov 03 19:16:47.283524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:47.283489 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" path="/var/lib/kubelet/pods/3afd215e-7b95-4ef7-b1d5-1ab1c58764e9/volumes" Nov 03 19:16:47.284034 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:47.284018 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" path="/var/lib/kubelet/pods/b665b35c-7648-4d19-b8d3-e723680182a6/volumes" Nov 03 19:16:47.284438 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:47.284425 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" path="/var/lib/kubelet/pods/cb291fbd-f786-4780-8fff-e07e380f99b9/volumes" Nov 03 19:16:47.781636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:47.781602 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:47.995478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:47.995438 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:48.209826 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.209793 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:48.579618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.579574 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9"] Nov 03 19:16:48.580283 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580263 5988 cpu_manager.go:414] "containerMap: removing container" podUID="1c89b584-f9f5-4ef6-8a1f-94586db8d09b" containerName="maas-api" Nov 03 19:16:48.580334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580285 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c89b584-f9f5-4ef6-8a1f-94586db8d09b" containerName="maas-api" Nov 03 19:16:48.580334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580302 5988 cpu_manager.go:414] "containerMap: removing container" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" Nov 03 19:16:48.580334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580307 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" Nov 03 19:16:48.580334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580318 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" Nov 03 19:16:48.580334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580323 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" Nov 03 19:16:48.580334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580330 5988 cpu_manager.go:414] "containerMap: removing container" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" Nov 03 19:16:48.580334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580334 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580342 5988 cpu_manager.go:414] "containerMap: removing container" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580348 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580362 5988 cpu_manager.go:414] "containerMap: removing container" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580367 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580375 5988 cpu_manager.go:414] "containerMap: removing container" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580379 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580386 5988 cpu_manager.go:414] "containerMap: removing container" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580390 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580399 5988 cpu_manager.go:414] "containerMap: removing container" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580403 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580410 5988 cpu_manager.go:414] "containerMap: removing container" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580414 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580424 5988 cpu_manager.go:414] "containerMap: removing container" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580431 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580444 5988 cpu_manager.go:414] "containerMap: removing container" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580448 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580455 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580459 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580469 5988 cpu_manager.go:414] "containerMap: removing container" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580473 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580483 5988 cpu_manager.go:414] "containerMap: removing container" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580489 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580497 5988 cpu_manager.go:414] "containerMap: removing container" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580502 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580515 5988 cpu_manager.go:414] "containerMap: removing container" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580521 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580530 5988 cpu_manager.go:414] "containerMap: removing container" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580535 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580541 5988 cpu_manager.go:414] "containerMap: removing container" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580545 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.580540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580551 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580555 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580562 5988 cpu_manager.go:414] "containerMap: removing container" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580567 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580576 5988 cpu_manager.go:414] "containerMap: removing container" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580580 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580586 5988 cpu_manager.go:414] "containerMap: removing container" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580592 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580601 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f119ec89-accf-40b3-87f7-dcf59d978d0f" containerName="maas-api" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580607 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f119ec89-accf-40b3-87f7-dcf59d978d0f" containerName="maas-api" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580615 5988 cpu_manager.go:414] "containerMap: removing container" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580619 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580626 5988 cpu_manager.go:414] "containerMap: removing container" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580630 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580637 5988 cpu_manager.go:414] "containerMap: removing container" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580641 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580649 5988 cpu_manager.go:414] "containerMap: removing container" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580654 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580664 5988 cpu_manager.go:414] "containerMap: removing container" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580670 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580793 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580803 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580808 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580818 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580824 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580833 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580842 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="10baada4-97eb-4266-b663-ebd01cd83a23" containerName="main" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580850 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580855 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580860 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580864 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580870 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580876 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580881 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580885 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f119ec89-accf-40b3-87f7-dcf59d978d0f" containerName="maas-api" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580892 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580897 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b665b35c-7648-4d19-b8d3-e723680182a6" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580904 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e6242f-3785-4ac5-818c-20f85285bc49" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580910 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580918 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580927 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f929aba5-31a4-4bd0-a2bd-afc3de03159a" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580933 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580939 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c89b584-f9f5-4ef6-8a1f-94586db8d09b" containerName="maas-api" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580946 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8de12aa-96f8-4e54-ad0d-6e5fa89202ce" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580951 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afd215e-7b95-4ef7-b1d5-1ab1c58764e9" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580957 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de86ec7-7982-4771-a8a8-c62921c86eb6" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580962 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cffe7d7-51e0-49f6-9acf-2cef2889878f" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580967 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb291fbd-f786-4780-8fff-e07e380f99b9" containerName="istio-proxy" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580971 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:16:48.581427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.580978 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="295e4c83-5d79-4f4a-9e3f-fac2c816f4b7" containerName="main" Nov 03 19:16:48.585648 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.585630 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:48.611819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.611792 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9"] Nov 03 19:16:48.626539 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.626508 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b89d93-3ebe-499e-9ed9-00770e4805c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:48.626742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.626665 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnrh\" (UniqueName: \"kubernetes.io/projected/c8b89d93-3ebe-499e-9ed9-00770e4805c3-kube-api-access-smnrh\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:48.672356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.655227 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3d74f801\"" Nov 03 19:16:48.701819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.701796 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3d74f801-dockercfg-pc5bq\"" Nov 03 19:16:48.727970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.727944 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnrh\" (UniqueName: \"kubernetes.io/projected/c8b89d93-3ebe-499e-9ed9-00770e4805c3-kube-api-access-smnrh\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:48.728113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.728099 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b89d93-3ebe-499e-9ed9-00770e4805c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:48.728805 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.728784 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b89d93-3ebe-499e-9ed9-00770e4805c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:48.785496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.785467 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnrh\" (UniqueName: \"kubernetes.io/projected/c8b89d93-3ebe-499e-9ed9-00770e4805c3-kube-api-access-smnrh\") pod \"payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:48.899657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:48.899582 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:49.182284 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:49.182203 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:49.311411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:49.311387 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9"] Nov 03 19:16:49.313951 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:16:49.313924 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b89d93_3ebe_499e_9ed9_00770e4805c3.slice/crio-d48d5f15beaeacf70cf27f8ef387aaa9af917643e927f2fc1c4cabd6a2ebbbcb WatchSource:0}: Error finding container d48d5f15beaeacf70cf27f8ef387aaa9af917643e927f2fc1c4cabd6a2ebbbcb: Status 404 returned error can't find the container with id d48d5f15beaeacf70cf27f8ef387aaa9af917643e927f2fc1c4cabd6a2ebbbcb Nov 03 19:16:49.394347 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:49.394317 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:49.437123 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:49.437038 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg"] Nov 03 19:16:49.437655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:49.437624 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" containerID="cri-o://17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39" gracePeriod=30 Nov 03 19:16:49.645717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:49.639355 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:50.000192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:50.000149 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" event={"ID":"c8b89d93-3ebe-499e-9ed9-00770e4805c3","Type":"ContainerStarted","Data":"de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419"} Nov 03 19:16:50.000623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:50.000210 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" event={"ID":"c8b89d93-3ebe-499e-9ed9-00770e4805c3","Type":"ContainerStarted","Data":"d48d5f15beaeacf70cf27f8ef387aaa9af917643e927f2fc1c4cabd6a2ebbbcb"} Nov 03 19:16:50.000623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:50.000233 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:16:50.116178 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:50.116121 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" podStartSLOduration=2.116104149 podStartE2EDuration="2.116104149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:50.114649629 +0000 UTC m=+2284.996065731" watchObservedRunningTime="2026-11-03 19:16:50.116104149 +0000 UTC m=+2284.997520252" Nov 03 19:16:50.571385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:50.571345 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:50.784083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:50.784049 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:51.085474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:51.085437 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:51.304928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:51.304896 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:52.237030 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:52.236987 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:53.893575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:53.893541 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:54.141145 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.141107 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:54.352181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.352144 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:54.405151 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.405126 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj"] Nov 03 19:16:54.450753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.450701 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba5674-9f8cj"] Nov 03 19:16:54.713995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.713970 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:16:54.738490 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738455 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-certs\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.738651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738506 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-envoy\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.738651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738530 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-socket\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.738651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738560 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-podinfo\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.738868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738702 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-credential-socket\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.738868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738732 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istiod-ca-cert\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.738868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738820 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-certs" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:54.739033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738902 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-token\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.739033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.738934 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-kube-api-access-7cnnv\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-kube-api-access-7cnnv\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.739132 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739059 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-data\") pod \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\" (UID: \"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7\") " Nov 03 19:16:54.739772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739205 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istiod-ca-cert" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:16:54.739772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739290 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-credential-socket" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:54.739772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739600 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-data" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:54.739772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739669 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-socket" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:54.739772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739709 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.739772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739734 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.739772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.739749 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.741522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.741493 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-envoy" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:16:54.741522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.741506 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-token" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:54.741863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.741844 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-kube-api-access-7cnnv" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "kube-api-access-7cnnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:16:54.741957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.741945 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-podinfo" pod "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" (UID: "f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:16:54.841396 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.841358 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.841584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.841404 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cnnv\" (UniqueName: \"kubernetes.io/projected/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-kube-api-access-7cnnv\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.841584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.841424 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.841584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.841438 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.841584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.841452 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.841584 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.841465 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:16:54.897738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:54.897649 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:55.137462 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.137432 5988 generic.go:356] "Generic (PLEG): container finished" podID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerID="17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39" exitCode=0 Nov 03 19:16:55.137623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.137480 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" event={"ID":"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7","Type":"ContainerDied","Data":"17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39"} Nov 03 19:16:55.137623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.137526 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" event={"ID":"f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7","Type":"ContainerDied","Data":"4ca89713e902626ea968dd6ccbfece35006b21ff6dd620e9d2e145e33f315889"} Nov 03 19:16:55.137623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.137548 5988 scope.go:122] "RemoveContainer" containerID="17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39" Nov 03 19:16:55.137623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.137473 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg" Nov 03 19:16:55.148867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.148849 5988 scope.go:122] "RemoveContainer" containerID="17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39" Nov 03 19:16:55.149142 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:16:55.149116 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39\": container with ID starting with 17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39 not found: ID does not exist" containerID="17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39" Nov 03 19:16:55.149208 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.149155 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39"} err="failed to get container status \"17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39\": rpc error: code = NotFound desc = could not find container \"17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39\": container with ID starting with 17ed36890e6277c61f14ca23c892b7db4d31ce164c9c17f4612af1c3db5a0b39 not found: ID does not exist" Nov 03 19:16:55.273311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.273278 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68654e8e-63aa-4547-bd5c-11e0be2ff68b" path="/var/lib/kubelet/pods/68654e8e-63aa-4547-bd5c-11e0be2ff68b/volumes" Nov 03 19:16:55.294016 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.293985 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg"] Nov 03 19:16:55.341619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:55.341582 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-fba266-openshift-default-5ff997597f-wfcwg"] Nov 03 19:16:55.937202 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:16:55.937164 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:16:56.145380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:56.145352 5988 generic.go:356] "Generic (PLEG): container finished" podID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerID="49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f" exitCode=0 Nov 03 19:16:56.146040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:56.145431 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" event={"ID":"807178f7-57ea-44bc-94ed-56e4f3ea3cae","Type":"ContainerDied","Data":"49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f"} Nov 03 19:16:56.146040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:56.145911 5988 scope.go:122] "RemoveContainer" containerID="49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f" Nov 03 19:16:56.147555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:56.147522 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:16:56.147678 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:56.147609 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:16:56.147678 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:56.147656 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:16:57.175787 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:57.175748 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" event={"ID":"807178f7-57ea-44bc-94ed-56e4f3ea3cae","Type":"ContainerStarted","Data":"3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407"} Nov 03 19:16:57.281109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:57.281075 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" path="/var/lib/kubelet/pods/f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7/volumes" Nov 03 19:16:57.469975 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:57.469874 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:16:57.470348 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:57.470253 5988 patch_prober.go:29] interesting pod/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:16:57.470555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:57.470529 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:16:58.471070 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:58.471028 5988 patch_prober.go:29] interesting pod/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:16:58.471611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:58.471094 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:16:59.006377 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:59.006346 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:16:59.471103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:59.471067 5988 patch_prober.go:29] interesting pod/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:16:59.471526 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:16:59.471122 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:17:00.468708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:00.468072 5988 patch_prober.go:29] interesting pod/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:17:00.468708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:00.468140 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:17:00.873708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:00.873397 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:01.086149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:01.086099 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:01.096256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:01.096204 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:17:01.298073 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:01.298000 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:01.472205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:01.472174 5988 patch_prober.go:29] interesting pod/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:17:01.472436 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:01.472415 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:17:02.466819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:02.466786 5988 patch_prober.go:29] interesting pod/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 19:17:02.467223 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:02.466851 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 19:17:03.471082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:03.471050 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:17:03.471560 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:03.471347 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:17:03.472190 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:03.472173 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:17:03.737509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:03.737419 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:08.515267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:08.515227 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:13.026392 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:13.026351 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:13.483524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:13.483487 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs"] Nov 03 19:17:13.841919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:13.841875 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:14.058666 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:14.058638 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:17.618276 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:17.618231 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:18.484255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:18.484218 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" podUID="330f5ff2-913a-4a92-bc49-7204c9480910" containerName="payload-pre-processing" containerID="cri-o://2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7" gracePeriod=25 Nov 03 19:17:18.758881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:18.758852 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:17:18.933638 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:18.933599 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330f5ff2-913a-4a92-bc49-7204c9480910-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330f5ff2-913a-4a92-bc49-7204c9480910-plugins-config-volume\") pod \"330f5ff2-913a-4a92-bc49-7204c9480910\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " Nov 03 19:17:18.933928 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:18.933754 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330f5ff2-913a-4a92-bc49-7204c9480910-kube-api-access-lshx2\" (UniqueName: \"kubernetes.io/projected/330f5ff2-913a-4a92-bc49-7204c9480910-kube-api-access-lshx2\") pod \"330f5ff2-913a-4a92-bc49-7204c9480910\" (UID: \"330f5ff2-913a-4a92-bc49-7204c9480910\") " Nov 03 19:17:18.934032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:18.934006 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330f5ff2-913a-4a92-bc49-7204c9480910-plugins-config-volume" pod "330f5ff2-913a-4a92-bc49-7204c9480910" (UID: "330f5ff2-913a-4a92-bc49-7204c9480910"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:18.934233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:18.934209 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330f5ff2-913a-4a92-bc49-7204c9480910-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:18.935963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:18.935933 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330f5ff2-913a-4a92-bc49-7204c9480910-kube-api-access-lshx2" pod "330f5ff2-913a-4a92-bc49-7204c9480910" (UID: "330f5ff2-913a-4a92-bc49-7204c9480910"). InnerVolumeSpecName "kube-api-access-lshx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:19.035597 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.035528 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lshx2\" (UniqueName: \"kubernetes.io/projected/330f5ff2-913a-4a92-bc49-7204c9480910-kube-api-access-lshx2\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:19.318787 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.318754 5988 generic.go:356] "Generic (PLEG): container finished" podID="330f5ff2-913a-4a92-bc49-7204c9480910" containerID="2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7" exitCode=0 Nov 03 19:17:19.318937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.318792 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" Nov 03 19:17:19.318937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.318836 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" event={"ID":"330f5ff2-913a-4a92-bc49-7204c9480910","Type":"ContainerDied","Data":"2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7"} Nov 03 19:17:19.318937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.318882 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs" event={"ID":"330f5ff2-913a-4a92-bc49-7204c9480910","Type":"ContainerDied","Data":"68969c12f331c4241c65bfcdedae270be0c608ce0557fddb08265808fe3c8356"} Nov 03 19:17:19.318937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.318903 5988 scope.go:122] "RemoveContainer" containerID="2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7" Nov 03 19:17:19.328812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.328795 5988 scope.go:122] "RemoveContainer" containerID="2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7" Nov 03 19:17:19.329098 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:17:19.329064 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7\": container with ID starting with 2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7 not found: ID does not exist" containerID="2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7" Nov 03 19:17:19.329150 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.329109 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7"} err="failed to get container status \"2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7\": rpc error: code = NotFound desc = could not find container \"2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7\": container with ID starting with 2aa977279ab837a755d6775f421e5a5abe9e5bf098d666ad5a087c5ea0e36fb7 not found: ID does not exist" Nov 03 19:17:19.474112 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.474083 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs"] Nov 03 19:17:19.520695 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:19.520654 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efd81c-76f4d9c555-hgzvs"] Nov 03 19:17:21.273151 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:21.273103 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330f5ff2-913a-4a92-bc49-7204c9480910" path="/var/lib/kubelet/pods/330f5ff2-913a-4a92-bc49-7204c9480910/volumes" Nov 03 19:17:22.115818 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:22.115777 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:24.979333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:24.979299 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:17:25.939465 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:17:25.939423 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:17:26.713345 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:26.713292 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:27.758706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:27.756777 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:17:27.758706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:27.756839 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:17:28.034126 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:28.034031 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:17:28.034311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:28.034127 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:17:28.720076 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:28.720039 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:28.929979 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:28.929949 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:31.340365 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:31.340326 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:35.867418 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:35.867372 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:38.128922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.128889 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546"] Nov 03 19:17:38.129453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129439 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:17:38.129498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129454 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eedf97f-8b5d-4c39-b931-76086a20b7fa" containerName="cleanup" Nov 03 19:17:38.129498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129468 5988 cpu_manager.go:414] "containerMap: removing container" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:17:38.129498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129473 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f32a5c-5c74-4495-8d14-3bfecfe81596" containerName="cleanup" Nov 03 19:17:38.129498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129487 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" Nov 03 19:17:38.129498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129492 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129511 5988 cpu_manager.go:414] "containerMap: removing container" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129516 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129523 5988 cpu_manager.go:414] "containerMap: removing container" podUID="330f5ff2-913a-4a92-bc49-7204c9480910" containerName="payload-pre-processing" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129528 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330f5ff2-913a-4a92-bc49-7204c9480910" containerName="payload-pre-processing" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129538 5988 cpu_manager.go:414] "containerMap: removing container" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129542 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129619 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f671f0-b432-4cb0-8c2c-d61d5d0d5da7" containerName="istio-proxy" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129629 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="330f5ff2-913a-4a92-bc49-7204c9480910" containerName="payload-pre-processing" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129639 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="344a6ecd-90de-4398-abff-9e246298d785" containerName="cleanup" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129647 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f3e8db-5cf4-4793-8771-05c8075a48f6" containerName="cleanup" Nov 03 19:17:38.129655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129654 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="0664c8b7-6cd4-443a-9802-a7fecc7cd186" containerName="cleanup" Nov 03 19:17:38.129995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.129662 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="453e6d0d-1386-4605-90e8-428474307891" containerName="cleanup" Nov 03 19:17:38.132509 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.132484 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.284970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.284935 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9779-3386-42a7-973c-db952983f870-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.284970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.284978 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.285180 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.284999 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2jfl\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-kube-api-access-n2jfl\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.285180 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.285082 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9779-3386-42a7-973c-db952983f870-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.285248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.285227 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.285305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.285292 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.285402 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.285388 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.285455 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.285443 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.285489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.285466 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.328795 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.328758 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546"] Nov 03 19:17:38.387209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387132 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387166 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387228 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9779-3386-42a7-973c-db952983f870-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387257 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387293 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2jfl\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-kube-api-access-n2jfl\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387331 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9779-3386-42a7-973c-db952983f870-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387393 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387479 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387557 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387596 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387727 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.387966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387758 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.388033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.387975 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.388138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.388117 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9779-3386-42a7-973c-db952983f870-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.389847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.389819 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.390054 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.390037 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9779-3386-42a7-973c-db952983f870-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.444447 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.444417 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.444610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.444584 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2jfl\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-kube-api-access-n2jfl\") pod \"maas-default-gateway-openshift-default-7854949c9-hm546\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.445392 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.445372 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:38.877471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.877414 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546"] Nov 03 19:17:38.880466 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:17:38.880429 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40f9779_3386_42a7_973c_db952983f870.slice/crio-3a1537c8a2bfc31c73abd7b55bdd7a6d9def18aa8d1cabc710ec82f6ae841c91 WatchSource:0}: Error finding container 3a1537c8a2bfc31c73abd7b55bdd7a6d9def18aa8d1cabc710ec82f6ae841c91: Status 404 returned error can't find the container with id 3a1537c8a2bfc31c73abd7b55bdd7a6d9def18aa8d1cabc710ec82f6ae841c91 Nov 03 19:17:38.883308 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.883261 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:17:38.883427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.883338 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:17:38.883427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:38.883381 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:17:39.487944 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:39.487900 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" event={"ID":"f40f9779-3386-42a7-973c-db952983f870","Type":"ContainerStarted","Data":"09cd176c85a5c1871facab6f0f4a35d6d561db21fdd014c93a10246bd868dff8"} Nov 03 19:17:39.488411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:39.487970 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" event={"ID":"f40f9779-3386-42a7-973c-db952983f870","Type":"ContainerStarted","Data":"3a1537c8a2bfc31c73abd7b55bdd7a6d9def18aa8d1cabc710ec82f6ae841c91"} Nov 03 19:17:39.603132 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:39.603077 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podStartSLOduration=1.6030608480000001 podStartE2EDuration="1.603060848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:39.60132696 +0000 UTC m=+2334.482743054" watchObservedRunningTime="2026-11-03 19:17:39.603060848 +0000 UTC m=+2334.484476951" Nov 03 19:17:40.466707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:40.462727 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:40.466707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:40.462854 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hm546 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:17:40.466707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:40.462897 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:17:40.684717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:40.683569 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:41.459708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:41.451008 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hm546 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:17:41.459708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:41.451070 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:17:42.467707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:42.451026 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hm546 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:17:42.467707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:42.451086 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:17:42.777764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:42.776637 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:43.244864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.244830 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:43.450709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.448052 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hm546 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:17:43.450709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.448116 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:17:43.807706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.806010 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg"] Nov 03 19:17:43.812006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.810621 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.861951 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.861918 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-942af8-openshift-default-dockercfg-lnck5\"" Nov 03 19:17:43.897466 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.897434 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-envoy\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.897746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.897729 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-token\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.897898 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.897880 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-podinfo\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.898022 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.898006 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-certs\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.898145 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.898130 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-data\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.898246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.898236 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-credential-socket\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.898327 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.898316 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-socket\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.898395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.898385 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpv2q\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-kube-api-access-hpv2q\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.898497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.898487 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67e964c-219b-43c9-9e23-0e1aa854fbff-istiod-ca-cert\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:43.911940 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.911911 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg"] Nov 03 19:17:43.997062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:43.997024 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:44.001611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.001581 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-token\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.001899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.001874 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-podinfo\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.002467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.002445 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-certs\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.002661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.002637 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-data\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.002870 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.002849 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-credential-socket\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003014 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.002999 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-socket\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003123 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.003109 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpv2q\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-kube-api-access-hpv2q\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.003270 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67e964c-219b-43c9-9e23-0e1aa854fbff-istiod-ca-cert\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.003129 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-credential-socket\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003539 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.003006 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-data\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003539 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.003289 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-socket\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003539 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.002940 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-certs\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.003841 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-envoy\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.003988 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.003967 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67e964c-219b-43c9-9e23-0e1aa854fbff-istiod-ca-cert\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.005231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.005206 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-podinfo\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.005789 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.005767 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-envoy\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.062746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.061613 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpv2q\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-kube-api-access-hpv2q\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.070706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.070281 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-token\") pod \"e2e-inf-b-942af8-openshift-default-878544c45-bpzgg\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.129178 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.129146 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:44.218801 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.218751 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:44.449876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.449838 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hm546 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:17:44.450196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.450146 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:17:44.600060 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.600026 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg"] Nov 03 19:17:44.601616 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:17:44.601586 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67e964c_219b_43c9_9e23_0e1aa854fbff.slice/crio-74d5e5c14450625bf7a40d06e6541f73ef84ecc9cea0a1195c46294f8c32704d WatchSource:0}: Error finding container 74d5e5c14450625bf7a40d06e6541f73ef84ecc9cea0a1195c46294f8c32704d: Status 404 returned error can't find the container with id 74d5e5c14450625bf7a40d06e6541f73ef84ecc9cea0a1195c46294f8c32704d Nov 03 19:17:44.604756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.604676 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:17:44.604848 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.604817 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:17:44.604899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:44.604864 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:17:45.445924 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:45.445884 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hm546 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:17:45.446412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:45.445955 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:17:45.530866 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:45.530826 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" event={"ID":"b67e964c-219b-43c9-9e23-0e1aa854fbff","Type":"ContainerStarted","Data":"1efeb9a34e39b9fc08ff33b8092f496221d04043f596a483c21091f06902a992"} Nov 03 19:17:45.531045 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:45.530881 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" event={"ID":"b67e964c-219b-43c9-9e23-0e1aa854fbff","Type":"ContainerStarted","Data":"74d5e5c14450625bf7a40d06e6541f73ef84ecc9cea0a1195c46294f8c32704d"} Nov 03 19:17:45.662469 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:45.662416 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podStartSLOduration=2.66240195 podStartE2EDuration="2.66240195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:45.66055516 +0000 UTC m=+2340.541971263" watchObservedRunningTime="2026-11-03 19:17:45.66240195 +0000 UTC m=+2340.543818052" Nov 03 19:17:46.129710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:46.129662 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:46.136272 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:46.136244 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:46.402065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:46.401934 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:46.446203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:46.446171 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hm546 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 19:17:46.446603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:46.446227 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 19:17:46.536654 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:46.536627 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:46.537675 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:46.537652 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:17:47.479516 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:47.479465 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:47.480045 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:47.480026 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:47.480945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:47.480924 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:17:48.597043 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:48.597013 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:48.817842 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:48.817806 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:49.036140 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:49.036030 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:50.721618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:50.721583 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb"] Nov 03 19:17:50.722080 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:50.721803 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" podUID="e25b294e-39c2-479d-b7eb-da2680d5a28c" containerName="maas-api" containerID="cri-o://22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418" gracePeriod=30 Nov 03 19:17:50.917270 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:50.917195 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:51.025434 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.025401 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:17:51.098196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.098158 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-metrics-certs\") pod \"e25b294e-39c2-479d-b7eb-da2680d5a28c\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " Nov 03 19:17:51.098723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.098226 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25b294e-39c2-479d-b7eb-da2680d5a28c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25b294e-39c2-479d-b7eb-da2680d5a28c-openshift-service-ca\") pod \"e25b294e-39c2-479d-b7eb-da2680d5a28c\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " Nov 03 19:17:51.098723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.098267 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25b294e-39c2-479d-b7eb-da2680d5a28c-kube-api-access-xx6ws\" (UniqueName: \"kubernetes.io/projected/e25b294e-39c2-479d-b7eb-da2680d5a28c-kube-api-access-xx6ws\") pod \"e25b294e-39c2-479d-b7eb-da2680d5a28c\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " Nov 03 19:17:51.098723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.098391 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls\") pod \"e25b294e-39c2-479d-b7eb-da2680d5a28c\" (UID: \"e25b294e-39c2-479d-b7eb-da2680d5a28c\") " Nov 03 19:17:51.098723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.098632 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25b294e-39c2-479d-b7eb-da2680d5a28c-openshift-service-ca" pod "e25b294e-39c2-479d-b7eb-da2680d5a28c" (UID: "e25b294e-39c2-479d-b7eb-da2680d5a28c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:17:51.098989 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.098970 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e25b294e-39c2-479d-b7eb-da2680d5a28c-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:51.100982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.100951 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls" pod "e25b294e-39c2-479d-b7eb-da2680d5a28c" (UID: "e25b294e-39c2-479d-b7eb-da2680d5a28c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:17:51.101079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.100998 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-metrics-certs" pod "e25b294e-39c2-479d-b7eb-da2680d5a28c" (UID: "e25b294e-39c2-479d-b7eb-da2680d5a28c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:17:51.101079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.101042 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25b294e-39c2-479d-b7eb-da2680d5a28c-kube-api-access-xx6ws" pod "e25b294e-39c2-479d-b7eb-da2680d5a28c" (UID: "e25b294e-39c2-479d-b7eb-da2680d5a28c"). InnerVolumeSpecName "kube-api-access-xx6ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:17:51.208514 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.208468 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:51.208514 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.208506 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e25b294e-39c2-479d-b7eb-da2680d5a28c-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:51.208846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.208526 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6ws\" (UniqueName: \"kubernetes.io/projected/e25b294e-39c2-479d-b7eb-da2680d5a28c-kube-api-access-xx6ws\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:17:51.575009 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.574972 5988 generic.go:356] "Generic (PLEG): container finished" podID="e25b294e-39c2-479d-b7eb-da2680d5a28c" containerID="22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418" exitCode=0 Nov 03 19:17:51.575186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.575023 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" Nov 03 19:17:51.575186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.575052 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" event={"ID":"e25b294e-39c2-479d-b7eb-da2680d5a28c","Type":"ContainerDied","Data":"22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418"} Nov 03 19:17:51.575186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.575108 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb" event={"ID":"e25b294e-39c2-479d-b7eb-da2680d5a28c","Type":"ContainerDied","Data":"60c6fb32a700aacb3ef35dcde8d3b4c14e2cdc431a65301da8e5ec4e738f8f46"} Nov 03 19:17:51.575186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.575132 5988 scope.go:122] "RemoveContainer" containerID="22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418" Nov 03 19:17:51.585248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.585223 5988 scope.go:122] "RemoveContainer" containerID="22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418" Nov 03 19:17:51.585555 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:17:51.585536 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418\": container with ID starting with 22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418 not found: ID does not exist" containerID="22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418" Nov 03 19:17:51.585641 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.585564 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418"} err="failed to get container status \"22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418\": rpc error: code = NotFound desc = could not find container \"22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418\": container with ID starting with 22a4c984eb7f17d63d5bf9752a69ebd9b79eb4cb8283348d1b291d774e694418 not found: ID does not exist" Nov 03 19:17:51.751539 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.751501 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb"] Nov 03 19:17:51.798266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.798236 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-65e151-65fddc644f-wjpjb"] Nov 03 19:17:51.976223 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:51.976147 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:52.183537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.183494 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:52.457559 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.457522 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4"] Nov 03 19:17:52.458171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.458155 5988 cpu_manager.go:414] "containerMap: removing container" podUID="e25b294e-39c2-479d-b7eb-da2680d5a28c" containerName="maas-api" Nov 03 19:17:52.458229 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.458171 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25b294e-39c2-479d-b7eb-da2680d5a28c" containerName="maas-api" Nov 03 19:17:52.458280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.458269 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25b294e-39c2-479d-b7eb-da2680d5a28c" containerName="maas-api" Nov 03 19:17:52.462026 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.462005 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:52.517060 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.516808 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-942af8-dockercfg-tltjc\"" Nov 03 19:17:52.517060 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.516920 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-942af8\"" Nov 03 19:17:52.547038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.539507 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fb572c-57ca-401c-8222-37dfb13aa9f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:52.547038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.539763 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmmm\" (UniqueName: \"kubernetes.io/projected/a0fb572c-57ca-401c-8222-37dfb13aa9f7-kube-api-access-rkmmm\") pod \"payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:52.547038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.542474 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4"] Nov 03 19:17:52.587901 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.587862 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:52.640669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.640637 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmmm\" (UniqueName: \"kubernetes.io/projected/a0fb572c-57ca-401c-8222-37dfb13aa9f7-kube-api-access-rkmmm\") pod \"payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:52.640865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.640784 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fb572c-57ca-401c-8222-37dfb13aa9f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:52.641552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.641525 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fb572c-57ca-401c-8222-37dfb13aa9f7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:52.700896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.700870 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmmm\" (UniqueName: \"kubernetes.io/projected/a0fb572c-57ca-401c-8222-37dfb13aa9f7-kube-api-access-rkmmm\") pod \"payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:52.778034 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:52.777960 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:53.067672 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.067636 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:53.150242 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.150205 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4"] Nov 03 19:17:53.272922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.272889 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25b294e-39c2-479d-b7eb-da2680d5a28c" path="/var/lib/kubelet/pods/e25b294e-39c2-479d-b7eb-da2680d5a28c/volumes" Nov 03 19:17:53.592769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.592732 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" event={"ID":"a0fb572c-57ca-401c-8222-37dfb13aa9f7","Type":"ContainerStarted","Data":"2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e"} Nov 03 19:17:53.592968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.592790 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" event={"ID":"a0fb572c-57ca-401c-8222-37dfb13aa9f7","Type":"ContainerStarted","Data":"c147ccf1208a7233d144bd06bb555ab6b5817b4fefbf4fa7e1285ba1ac590496"} Nov 03 19:17:53.592968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.592829 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:17:53.701860 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.701801 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" podStartSLOduration=1.701781892 podStartE2EDuration="1.701781892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:17:53.700874804 +0000 UTC m=+2348.582290905" watchObservedRunningTime="2026-11-03 19:17:53.701781892 +0000 UTC m=+2348.583197995" Nov 03 19:17:53.716714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.716673 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:53.924719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:53.924553 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:54.223851 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:54.223729 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:54.424229 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:54.424196 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:54.629598 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:54.629562 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:54.898807 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:54.898721 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:55.937260 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:17:55.937220 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:17:59.074419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:59.074342 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:59.280968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:59.280925 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:17:59.553332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:17:59.553248 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:02.018920 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:02.018880 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:02.686692 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:02.686645 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:03.062358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.062265 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:03.116978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.116948 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7"] Nov 03 19:18:03.120993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.120970 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.180398 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.180372 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctvp\" (UniqueName: \"kubernetes.io/projected/c51a992e-7a43-4b7d-9765-5a1741254d80-kube-api-access-vctvp\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.180525 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.180428 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.180572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.180554 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.216693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.216640 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:18:03.216693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.216660 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:18:03.219254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.219232 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7"] Nov 03 19:18:03.282065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.282026 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.282251 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.282196 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctvp\" (UniqueName: \"kubernetes.io/projected/c51a992e-7a43-4b7d-9765-5a1741254d80-kube-api-access-vctvp\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.282334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.282318 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.282781 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.282757 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.284544 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.284521 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.341776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.341751 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctvp\" (UniqueName: \"kubernetes.io/projected/c51a992e-7a43-4b7d-9765-5a1741254d80-kube-api-access-vctvp\") pod \"maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.434100 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.434072 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:03.802782 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:03.802755 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7"] Nov 03 19:18:03.805349 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:18:03.805310 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51a992e_7a43_4b7d_9765_5a1741254d80.slice/crio-6f97c01e677a73dbb6b802a23ee0f78f165accb9e2dc974974bc20e9b694f588 WatchSource:0}: Error finding container 6f97c01e677a73dbb6b802a23ee0f78f165accb9e2dc974974bc20e9b694f588: Status 404 returned error can't find the container with id 6f97c01e677a73dbb6b802a23ee0f78f165accb9e2dc974974bc20e9b694f588 Nov 03 19:18:04.340884 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:04.340847 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:04.602341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:04.602267 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:18:04.677392 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:04.677358 5988 generic.go:356] "Generic (PLEG): container finished" podID="c51a992e-7a43-4b7d-9765-5a1741254d80" containerID="cabab46b3e4cc218c61efdb529d443d5d317fbe3ef9923fa73359ea766450eef" exitCode=0 Nov 03 19:18:04.677533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:04.677397 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" event={"ID":"c51a992e-7a43-4b7d-9765-5a1741254d80","Type":"ContainerDied","Data":"cabab46b3e4cc218c61efdb529d443d5d317fbe3ef9923fa73359ea766450eef"} Nov 03 19:18:04.677533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:04.677455 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" event={"ID":"c51a992e-7a43-4b7d-9765-5a1741254d80","Type":"ContainerStarted","Data":"6f97c01e677a73dbb6b802a23ee0f78f165accb9e2dc974974bc20e9b694f588"} Nov 03 19:18:05.865458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.865419 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:05.913501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.913469 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-xks-ca\") pod \"c51a992e-7a43-4b7d-9765-5a1741254d80\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " Nov 03 19:18:05.913702 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.913538 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-openshift-ca\") pod \"c51a992e-7a43-4b7d-9765-5a1741254d80\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " Nov 03 19:18:05.913702 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.913644 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51a992e-7a43-4b7d-9765-5a1741254d80-kube-api-access-vctvp\" (UniqueName: \"kubernetes.io/projected/c51a992e-7a43-4b7d-9765-5a1741254d80-kube-api-access-vctvp\") pod \"c51a992e-7a43-4b7d-9765-5a1741254d80\" (UID: \"c51a992e-7a43-4b7d-9765-5a1741254d80\") " Nov 03 19:18:05.913969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.913942 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-openshift-ca" pod "c51a992e-7a43-4b7d-9765-5a1741254d80" (UID: "c51a992e-7a43-4b7d-9765-5a1741254d80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:05.914261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.914238 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:05.915725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.915672 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-xks-ca" pod "c51a992e-7a43-4b7d-9765-5a1741254d80" (UID: "c51a992e-7a43-4b7d-9765-5a1741254d80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:05.915811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:05.915787 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51a992e-7a43-4b7d-9765-5a1741254d80-kube-api-access-vctvp" pod "c51a992e-7a43-4b7d-9765-5a1741254d80" (UID: "c51a992e-7a43-4b7d-9765-5a1741254d80"). InnerVolumeSpecName "kube-api-access-vctvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:06.016024 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:06.015952 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51a992e-7a43-4b7d-9765-5a1741254d80-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.016024 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:06.015984 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctvp\" (UniqueName: \"kubernetes.io/projected/c51a992e-7a43-4b7d-9765-5a1741254d80-kube-api-access-vctvp\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:06.699213 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:06.699182 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" Nov 03 19:18:06.699429 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:06.699212 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7" event={"ID":"c51a992e-7a43-4b7d-9765-5a1741254d80","Type":"ContainerDied","Data":"6f97c01e677a73dbb6b802a23ee0f78f165accb9e2dc974974bc20e9b694f588"} Nov 03 19:18:06.699429 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:06.699257 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f97c01e677a73dbb6b802a23ee0f78f165accb9e2dc974974bc20e9b694f588" Nov 03 19:18:07.603474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:07.603440 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2"] Nov 03 19:18:07.746133 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:07.746080 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:09.272457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:09.272421 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:13.814368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:13.814305 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:14.357603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:14.357547 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:14.565667 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:14.565618 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:16.945241 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:16.945197 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:17.156310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:17.156269 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:17.799113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:17.799078 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:19.280186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:19.280150 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:19.513950 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:19.513916 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:19.793919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:19.793873 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:20.192853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:20.192813 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:21.331209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:21.331179 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:22.573239 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.573195 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:22.624532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.624483 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" containerID="cri-o://95e957d09fc563019f49ee7a570e37febcafa9d829438702dde2ef8666cdcc62" gracePeriod=45 Nov 03 19:18:22.775659 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.775626 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:22.816440 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.816406 5988 generic.go:356] "Generic (PLEG): container finished" podID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerID="95e957d09fc563019f49ee7a570e37febcafa9d829438702dde2ef8666cdcc62" exitCode=0 Nov 03 19:18:22.816622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.816482 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" event={"ID":"23929892-bb70-4d89-9d34-fbeb52b9759f","Type":"ContainerDied","Data":"95e957d09fc563019f49ee7a570e37febcafa9d829438702dde2ef8666cdcc62"} Nov 03 19:18:22.884410 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.884379 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:18:22.951515 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951486 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-home\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-home\") pod \"23929892-bb70-4d89-9d34-fbeb52b9759f\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " Nov 03 19:18:22.951709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951594 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23929892-bb70-4d89-9d34-fbeb52b9759f-kube-api-access-2cg5x\" (UniqueName: \"kubernetes.io/projected/23929892-bb70-4d89-9d34-fbeb52b9759f-kube-api-access-2cg5x\") pod \"23929892-bb70-4d89-9d34-fbeb52b9759f\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " Nov 03 19:18:22.951709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951622 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-tmp-dir\") pod \"23929892-bb70-4d89-9d34-fbeb52b9759f\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " Nov 03 19:18:22.951847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951717 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-dshm\") pod \"23929892-bb70-4d89-9d34-fbeb52b9759f\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " Nov 03 19:18:22.951847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951740 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-home" pod "23929892-bb70-4d89-9d34-fbeb52b9759f" (UID: "23929892-bb70-4d89-9d34-fbeb52b9759f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:22.951847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951751 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-model-cache\") pod \"23929892-bb70-4d89-9d34-fbeb52b9759f\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " Nov 03 19:18:22.951847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951799 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23929892-bb70-4d89-9d34-fbeb52b9759f-tls-certs\" (UniqueName: \"kubernetes.io/secret/23929892-bb70-4d89-9d34-fbeb52b9759f-tls-certs\") pod \"23929892-bb70-4d89-9d34-fbeb52b9759f\" (UID: \"23929892-bb70-4d89-9d34-fbeb52b9759f\") " Nov 03 19:18:22.952064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951964 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-model-cache" pod "23929892-bb70-4d89-9d34-fbeb52b9759f" (UID: "23929892-bb70-4d89-9d34-fbeb52b9759f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:22.952064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.951982 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-tmp-dir" pod "23929892-bb70-4d89-9d34-fbeb52b9759f" (UID: "23929892-bb70-4d89-9d34-fbeb52b9759f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:22.952383 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.952361 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:22.952523 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.952406 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:22.952523 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.952420 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:22.953966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.953932 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23929892-bb70-4d89-9d34-fbeb52b9759f-tls-certs" pod "23929892-bb70-4d89-9d34-fbeb52b9759f" (UID: "23929892-bb70-4d89-9d34-fbeb52b9759f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:22.953966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.953947 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-dshm" pod "23929892-bb70-4d89-9d34-fbeb52b9759f" (UID: "23929892-bb70-4d89-9d34-fbeb52b9759f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:22.954130 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:22.954067 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23929892-bb70-4d89-9d34-fbeb52b9759f-kube-api-access-2cg5x" pod "23929892-bb70-4d89-9d34-fbeb52b9759f" (UID: "23929892-bb70-4d89-9d34-fbeb52b9759f"). InnerVolumeSpecName "kube-api-access-2cg5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:23.053817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.053774 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cg5x\" (UniqueName: \"kubernetes.io/projected/23929892-bb70-4d89-9d34-fbeb52b9759f-kube-api-access-2cg5x\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.053817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.053809 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23929892-bb70-4d89-9d34-fbeb52b9759f-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.053817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.053824 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23929892-bb70-4d89-9d34-fbeb52b9759f-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:23.664984 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.664948 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn"] Nov 03 19:18:23.665508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.665492 5988 cpu_manager.go:414] "containerMap: removing container" podUID="c51a992e-7a43-4b7d-9765-5a1741254d80" containerName="revoke-keys" Nov 03 19:18:23.665571 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.665510 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51a992e-7a43-4b7d-9765-5a1741254d80" containerName="revoke-keys" Nov 03 19:18:23.665571 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.665566 5988 cpu_manager.go:414] "containerMap: removing container" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" Nov 03 19:18:23.665639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.665574 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" Nov 03 19:18:23.665783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.665768 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" Nov 03 19:18:23.665839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.665794 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51a992e-7a43-4b7d-9765-5a1741254d80" containerName="revoke-keys" Nov 03 19:18:23.669944 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.669924 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.725909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.725872 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn"] Nov 03 19:18:23.762668 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.762629 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzh4g\" (UniqueName: \"kubernetes.io/projected/82576b28-3cd0-4ebc-be44-195e748db46b-kube-api-access-wzh4g\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.762858 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.762710 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.762951 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.762919 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.769294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.769272 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:18:23.769404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.769301 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:18:23.804238 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.804199 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 19:18:23.823729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.823704 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" Nov 03 19:18:23.823729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.823720 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2" event={"ID":"23929892-bb70-4d89-9d34-fbeb52b9759f","Type":"ContainerDied","Data":"2d8e4618b6daa01987f1e496cf087ed73ab3cf04826db2ce67728e363660bb67"} Nov 03 19:18:23.823930 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.823769 5988 scope.go:122] "RemoveContainer" containerID="95e957d09fc563019f49ee7a570e37febcafa9d829438702dde2ef8666cdcc62" Nov 03 19:18:23.863621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.863594 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzh4g\" (UniqueName: \"kubernetes.io/projected/82576b28-3cd0-4ebc-be44-195e748db46b-kube-api-access-wzh4g\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.863778 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.863700 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.863846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.863833 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.864621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.864591 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.866079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.866059 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.920965 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.920911 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzh4g\" (UniqueName: \"kubernetes.io/projected/82576b28-3cd0-4ebc-be44-195e748db46b-kube-api-access-wzh4g\") pod \"maas-api-revoke-sub-a76ee201-c72zn\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.977098 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.977075 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2"] Nov 03 19:18:23.983717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.983698 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:23.985286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:23.985269 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-3a7056/simulator-w5-3a7056-kserve-6dcbdf5b68-htdk2"] Nov 03 19:18:24.356447 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:24.356412 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn"] Nov 03 19:18:24.360380 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:18:24.360349 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82576b28_3cd0_4ebc_be44_195e748db46b.slice/crio-0e172a7c77f3f174200768a0b275db3fc4f1a9959317a963aa5e783d0742a6d3 WatchSource:0}: Error finding container 0e172a7c77f3f174200768a0b275db3fc4f1a9959317a963aa5e783d0742a6d3: Status 404 returned error can't find the container with id 0e172a7c77f3f174200768a0b275db3fc4f1a9959317a963aa5e783d0742a6d3 Nov 03 19:18:24.608848 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:24.608769 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:24.833142 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:24.833103 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" event={"ID":"82576b28-3cd0-4ebc-be44-195e748db46b","Type":"ContainerStarted","Data":"7384142c3174f9ca6d8fd2d809316bea44b20069f06bb4863a0f690adf98885f"} Nov 03 19:18:24.833849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:24.833162 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" event={"ID":"82576b28-3cd0-4ebc-be44-195e748db46b","Type":"ContainerStarted","Data":"0e172a7c77f3f174200768a0b275db3fc4f1a9959317a963aa5e783d0742a6d3"} Nov 03 19:18:25.273351 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:25.273322 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23929892-bb70-4d89-9d34-fbeb52b9759f" path="/var/lib/kubelet/pods/23929892-bb70-4d89-9d34-fbeb52b9759f/volumes" Nov 03 19:18:25.841500 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:25.841462 5988 generic.go:356] "Generic (PLEG): container finished" podID="82576b28-3cd0-4ebc-be44-195e748db46b" containerID="7384142c3174f9ca6d8fd2d809316bea44b20069f06bb4863a0f690adf98885f" exitCode=0 Nov 03 19:18:25.841998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:25.841550 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" event={"ID":"82576b28-3cd0-4ebc-be44-195e748db46b","Type":"ContainerDied","Data":"7384142c3174f9ca6d8fd2d809316bea44b20069f06bb4863a0f690adf98885f"} Nov 03 19:18:25.939666 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:18:25.939630 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:18:26.015899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:26.015865 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:26.235196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:26.235115 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:26.261481 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:26.261451 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl"] Nov 03 19:18:26.460730 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:26.459846 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:26.678963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:26.678926 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:26.911585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:26.911543 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:27.011972 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.011946 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:27.103056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.103024 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-openshift-ca\") pod \"82576b28-3cd0-4ebc-be44-195e748db46b\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " Nov 03 19:18:27.103217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.103175 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82576b28-3cd0-4ebc-be44-195e748db46b-kube-api-access-wzh4g\" (UniqueName: \"kubernetes.io/projected/82576b28-3cd0-4ebc-be44-195e748db46b-kube-api-access-wzh4g\") pod \"82576b28-3cd0-4ebc-be44-195e748db46b\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " Nov 03 19:18:27.103217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.103203 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-xks-ca\") pod \"82576b28-3cd0-4ebc-be44-195e748db46b\" (UID: \"82576b28-3cd0-4ebc-be44-195e748db46b\") " Nov 03 19:18:27.103395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.103372 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-openshift-ca" pod "82576b28-3cd0-4ebc-be44-195e748db46b" (UID: "82576b28-3cd0-4ebc-be44-195e748db46b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:27.103814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.103794 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:27.105309 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.105280 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-xks-ca" pod "82576b28-3cd0-4ebc-be44-195e748db46b" (UID: "82576b28-3cd0-4ebc-be44-195e748db46b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:27.105415 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.105329 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82576b28-3cd0-4ebc-be44-195e748db46b-kube-api-access-wzh4g" pod "82576b28-3cd0-4ebc-be44-195e748db46b" (UID: "82576b28-3cd0-4ebc-be44-195e748db46b"). InnerVolumeSpecName "kube-api-access-wzh4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:27.204994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.204958 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzh4g\" (UniqueName: \"kubernetes.io/projected/82576b28-3cd0-4ebc-be44-195e748db46b-kube-api-access-wzh4g\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:27.204994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.204989 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82576b28-3cd0-4ebc-be44-195e748db46b-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:27.309082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.309055 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv"] Nov 03 19:18:27.309722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.309703 5988 cpu_manager.go:414] "containerMap: removing container" podUID="82576b28-3cd0-4ebc-be44-195e748db46b" containerName="revoke-keys" Nov 03 19:18:27.309722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.309721 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82576b28-3cd0-4ebc-be44-195e748db46b" containerName="revoke-keys" Nov 03 19:18:27.309901 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.309876 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="82576b28-3cd0-4ebc-be44-195e748db46b" containerName="revoke-keys" Nov 03 19:18:27.313340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.313321 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.367585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.367556 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv"] Nov 03 19:18:27.408222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.408194 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grrn\" (UniqueName: \"kubernetes.io/projected/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-kube-api-access-8grrn\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.408338 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.408321 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.408438 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.408420 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.510116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.510061 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grrn\" (UniqueName: \"kubernetes.io/projected/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-kube-api-access-8grrn\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.510294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.510151 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.510294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.510234 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.510990 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.510964 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.512541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.512522 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.567733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.567705 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grrn\" (UniqueName: \"kubernetes.io/projected/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-kube-api-access-8grrn\") pod \"maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.626964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.626929 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:27.854473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.854398 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" Nov 03 19:18:27.854473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.854436 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn" event={"ID":"82576b28-3cd0-4ebc-be44-195e748db46b","Type":"ContainerDied","Data":"0e172a7c77f3f174200768a0b275db3fc4f1a9959317a963aa5e783d0742a6d3"} Nov 03 19:18:27.854659 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.854484 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e172a7c77f3f174200768a0b275db3fc4f1a9959317a963aa5e783d0742a6d3" Nov 03 19:18:27.996971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:27.996940 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv"] Nov 03 19:18:27.999338 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:18:27.999305 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded35ebff_edce_48ce_8eb1_fdc2304a1a94.slice/crio-c705eafe7754ca8b96f4adf79ff40c61d8a6491d3636ef7f76d27c00831c0dab WatchSource:0}: Error finding container c705eafe7754ca8b96f4adf79ff40c61d8a6491d3636ef7f76d27c00831c0dab: Status 404 returned error can't find the container with id c705eafe7754ca8b96f4adf79ff40c61d8a6491d3636ef7f76d27c00831c0dab Nov 03 19:18:28.862279 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:28.862252 5988 generic.go:356] "Generic (PLEG): container finished" podID="ed35ebff-edce-48ce-8eb1-fdc2304a1a94" containerID="9f3fc9e048460c54b385b6d47805cefd0e99f082b3c160786f0082c55f8a3dd4" exitCode=0 Nov 03 19:18:28.862414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:28.862340 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" event={"ID":"ed35ebff-edce-48ce-8eb1-fdc2304a1a94","Type":"ContainerDied","Data":"9f3fc9e048460c54b385b6d47805cefd0e99f082b3c160786f0082c55f8a3dd4"} Nov 03 19:18:28.862414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:28.862391 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" event={"ID":"ed35ebff-edce-48ce-8eb1-fdc2304a1a94","Type":"ContainerStarted","Data":"c705eafe7754ca8b96f4adf79ff40c61d8a6491d3636ef7f76d27c00831c0dab"} Nov 03 19:18:29.320380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:29.320347 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:30.015888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.015863 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:30.143155 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.143077 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-xks-ca\") pod \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " Nov 03 19:18:30.143413 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.143295 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-kube-api-access-8grrn\" (UniqueName: \"kubernetes.io/projected/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-kube-api-access-8grrn\") pod \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " Nov 03 19:18:30.143413 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.143321 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-openshift-ca\") pod \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\" (UID: \"ed35ebff-edce-48ce-8eb1-fdc2304a1a94\") " Nov 03 19:18:30.143830 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.143798 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-openshift-ca" pod "ed35ebff-edce-48ce-8eb1-fdc2304a1a94" (UID: "ed35ebff-edce-48ce-8eb1-fdc2304a1a94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:30.145267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.145239 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-xks-ca" pod "ed35ebff-edce-48ce-8eb1-fdc2304a1a94" (UID: "ed35ebff-edce-48ce-8eb1-fdc2304a1a94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:30.145494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.145472 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-kube-api-access-8grrn" pod "ed35ebff-edce-48ce-8eb1-fdc2304a1a94" (UID: "ed35ebff-edce-48ce-8eb1-fdc2304a1a94"). InnerVolumeSpecName "kube-api-access-8grrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:30.245225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.245198 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grrn\" (UniqueName: \"kubernetes.io/projected/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-kube-api-access-8grrn\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:30.245225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.245223 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:30.245411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.245233 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed35ebff-edce-48ce-8eb1-fdc2304a1a94-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:30.875712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.875670 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" Nov 03 19:18:30.876108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.875710 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv" event={"ID":"ed35ebff-edce-48ce-8eb1-fdc2304a1a94","Type":"ContainerDied","Data":"c705eafe7754ca8b96f4adf79ff40c61d8a6491d3636ef7f76d27c00831c0dab"} Nov 03 19:18:30.876108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:30.875760 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c705eafe7754ca8b96f4adf79ff40c61d8a6491d3636ef7f76d27c00831c0dab" Nov 03 19:18:31.817545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:31.817507 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw"] Nov 03 19:18:32.027975 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:32.027938 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:33.807887 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:33.807831 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:34.015612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:34.015578 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:34.219953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:34.219918 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:34.724417 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:34.724379 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:37.154906 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:37.154864 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:18:39.404716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:39.403164 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:39.639600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:39.639542 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:40.194711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:40.192837 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:18:41.290192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.290145 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" containerID="cri-o://98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9" gracePeriod=45 Nov 03 19:18:41.566528 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.566499 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 19:18:41.687913 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.687877 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-home\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-home\") pod \"b0f79162-e784-4c53-af8d-e7524a5cc763\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " Nov 03 19:18:41.688104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688052 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f79162-e784-4c53-af8d-e7524a5cc763-kube-api-access-wpfjr\" (UniqueName: \"kubernetes.io/projected/b0f79162-e784-4c53-af8d-e7524a5cc763-kube-api-access-wpfjr\") pod \"b0f79162-e784-4c53-af8d-e7524a5cc763\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " Nov 03 19:18:41.688183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688160 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0f79162-e784-4c53-af8d-e7524a5cc763-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f79162-e784-4c53-af8d-e7524a5cc763-tls-certs\") pod \"b0f79162-e784-4c53-af8d-e7524a5cc763\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " Nov 03 19:18:41.688242 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688184 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-home" pod "b0f79162-e784-4c53-af8d-e7524a5cc763" (UID: "b0f79162-e784-4c53-af8d-e7524a5cc763"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:41.688242 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688189 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-dshm\") pod \"b0f79162-e784-4c53-af8d-e7524a5cc763\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " Nov 03 19:18:41.688355 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688285 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-model-cache\") pod \"b0f79162-e784-4c53-af8d-e7524a5cc763\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " Nov 03 19:18:41.688355 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688335 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-tmp-dir\") pod \"b0f79162-e784-4c53-af8d-e7524a5cc763\" (UID: \"b0f79162-e784-4c53-af8d-e7524a5cc763\") " Nov 03 19:18:41.688739 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688707 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-model-cache" pod "b0f79162-e784-4c53-af8d-e7524a5cc763" (UID: "b0f79162-e784-4c53-af8d-e7524a5cc763"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:41.689041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.688923 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-tmp-dir" pod "b0f79162-e784-4c53-af8d-e7524a5cc763" (UID: "b0f79162-e784-4c53-af8d-e7524a5cc763"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:41.689391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.689366 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:41.689487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.689396 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:41.689487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.689411 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:41.690834 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.690802 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-dshm" pod "b0f79162-e784-4c53-af8d-e7524a5cc763" (UID: "b0f79162-e784-4c53-af8d-e7524a5cc763"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:41.690942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.690825 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f79162-e784-4c53-af8d-e7524a5cc763-kube-api-access-wpfjr" pod "b0f79162-e784-4c53-af8d-e7524a5cc763" (UID: "b0f79162-e784-4c53-af8d-e7524a5cc763"). InnerVolumeSpecName "kube-api-access-wpfjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:41.691622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.691596 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f79162-e784-4c53-af8d-e7524a5cc763-tls-certs" pod "b0f79162-e784-4c53-af8d-e7524a5cc763" (UID: "b0f79162-e784-4c53-af8d-e7524a5cc763"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:41.790700 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.790640 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f79162-e784-4c53-af8d-e7524a5cc763-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:41.790700 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.790700 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f79162-e784-4c53-af8d-e7524a5cc763-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:41.790975 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:41.790725 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpfjr\" (UniqueName: \"kubernetes.io/projected/b0f79162-e784-4c53-af8d-e7524a5cc763-kube-api-access-wpfjr\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:42.383253 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.383209 5988 generic.go:356] "Generic (PLEG): container finished" podID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerID="98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9" exitCode=0 Nov 03 19:18:42.383814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.383271 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" Nov 03 19:18:42.383814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.383297 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" event={"ID":"b0f79162-e784-4c53-af8d-e7524a5cc763","Type":"ContainerDied","Data":"98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9"} Nov 03 19:18:42.383814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.383345 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl" event={"ID":"b0f79162-e784-4c53-af8d-e7524a5cc763","Type":"ContainerDied","Data":"0dbe644c4cb480e0d237fbf93818363af5c0041271dc569f34a8db336a2d0370"} Nov 03 19:18:42.383814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.383366 5988 scope.go:122] "RemoveContainer" containerID="98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9" Nov 03 19:18:42.395312 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.395289 5988 scope.go:122] "RemoveContainer" containerID="98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9" Nov 03 19:18:42.395602 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:18:42.395583 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9\": container with ID starting with 98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9 not found: ID does not exist" containerID="98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9" Nov 03 19:18:42.395671 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.395614 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9"} err="failed to get container status \"98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9\": rpc error: code = NotFound desc = could not find container \"98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9\": container with ID starting with 98a8182764618a0352d90310352842181c34ae4ecfd0e78d801914d94c5540d9 not found: ID does not exist" Nov 03 19:18:42.537877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.537841 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl"] Nov 03 19:18:42.588853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:42.586834 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9686c3/premium-w0-9686c3-kserve-6f89d849b7-h6bjl"] Nov 03 19:18:43.273497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:43.273458 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" path="/var/lib/kubelet/pods/b0f79162-e784-4c53-af8d-e7524a5cc763/volumes" Nov 03 19:18:44.275156 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:44.275118 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:44.479968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:44.479927 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:45.071361 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:45.071328 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:45.834943 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:45.834908 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:46.839259 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:46.839213 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" containerID="cri-o://34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd" gracePeriod=45 Nov 03 19:18:47.106091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.106026 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 19:18:47.257585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.257548 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-model-cache\") pod \"856af342-3dc8-4538-a6a0-e779029486a7\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " Nov 03 19:18:47.257822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.257638 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856af342-3dc8-4538-a6a0-e779029486a7-kube-api-access-99xhj\" (UniqueName: \"kubernetes.io/projected/856af342-3dc8-4538-a6a0-e779029486a7-kube-api-access-99xhj\") pod \"856af342-3dc8-4538-a6a0-e779029486a7\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " Nov 03 19:18:47.257822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.257673 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-home\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-home\") pod \"856af342-3dc8-4538-a6a0-e779029486a7\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " Nov 03 19:18:47.257822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.257723 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-dshm\") pod \"856af342-3dc8-4538-a6a0-e779029486a7\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " Nov 03 19:18:47.257822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.257765 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856af342-3dc8-4538-a6a0-e779029486a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/856af342-3dc8-4538-a6a0-e779029486a7-tls-certs\") pod \"856af342-3dc8-4538-a6a0-e779029486a7\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " Nov 03 19:18:47.258040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.257882 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-tmp-dir\") pod \"856af342-3dc8-4538-a6a0-e779029486a7\" (UID: \"856af342-3dc8-4538-a6a0-e779029486a7\") " Nov 03 19:18:47.258673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.258644 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-home" pod "856af342-3dc8-4538-a6a0-e779029486a7" (UID: "856af342-3dc8-4538-a6a0-e779029486a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.258888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.258738 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-tmp-dir" pod "856af342-3dc8-4538-a6a0-e779029486a7" (UID: "856af342-3dc8-4538-a6a0-e779029486a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.258888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.258846 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-model-cache" pod "856af342-3dc8-4538-a6a0-e779029486a7" (UID: "856af342-3dc8-4538-a6a0-e779029486a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.261467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.261440 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-dshm" pod "856af342-3dc8-4538-a6a0-e779029486a7" (UID: "856af342-3dc8-4538-a6a0-e779029486a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:18:47.261562 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.261497 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856af342-3dc8-4538-a6a0-e779029486a7-kube-api-access-99xhj" pod "856af342-3dc8-4538-a6a0-e779029486a7" (UID: "856af342-3dc8-4538-a6a0-e779029486a7"). InnerVolumeSpecName "kube-api-access-99xhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:47.261613 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.261575 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856af342-3dc8-4538-a6a0-e779029486a7-tls-certs" pod "856af342-3dc8-4538-a6a0-e779029486a7" (UID: "856af342-3dc8-4538-a6a0-e779029486a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:47.359435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.359358 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.359435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.359388 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xhj\" (UniqueName: \"kubernetes.io/projected/856af342-3dc8-4538-a6a0-e779029486a7-kube-api-access-99xhj\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.359435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.359398 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.359435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.359407 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.359435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.359416 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/856af342-3dc8-4538-a6a0-e779029486a7-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.359435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.359423 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/856af342-3dc8-4538-a6a0-e779029486a7-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:47.439722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.432933 5988 generic.go:356] "Generic (PLEG): container finished" podID="856af342-3dc8-4538-a6a0-e779029486a7" containerID="34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd" exitCode=0 Nov 03 19:18:47.439722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.432975 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" event={"ID":"856af342-3dc8-4538-a6a0-e779029486a7","Type":"ContainerDied","Data":"34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd"} Nov 03 19:18:47.439722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.433030 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" event={"ID":"856af342-3dc8-4538-a6a0-e779029486a7","Type":"ContainerDied","Data":"351b9a9ba107d9b927aa179798598ad141765d98fcb34c9691162d1678b5a20a"} Nov 03 19:18:47.439722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.433054 5988 scope.go:122] "RemoveContainer" containerID="34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd" Nov 03 19:18:47.439722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.432983 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw" Nov 03 19:18:47.444498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.444477 5988 scope.go:122] "RemoveContainer" containerID="34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd" Nov 03 19:18:47.444872 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:18:47.444851 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd\": container with ID starting with 34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd not found: ID does not exist" containerID="34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd" Nov 03 19:18:47.444957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.444883 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd"} err="failed to get container status \"34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd\": rpc error: code = NotFound desc = could not find container \"34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd\": container with ID starting with 34c63f1c40ac7d7d2c0d25a591540972df4da0a189e3358d3495602354f3c8bd not found: ID does not exist" Nov 03 19:18:47.588870 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.588835 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw"] Nov 03 19:18:47.591854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.591828 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a4cacd/premium-w3-a4cacd-kserve-bc459cfd9-zkkkw"] Nov 03 19:18:47.762033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.761950 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:47.981010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:47.980969 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:49.277068 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.277029 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856af342-3dc8-4538-a6a0-e779029486a7" path="/var/lib/kubelet/pods/856af342-3dc8-4538-a6a0-e779029486a7/volumes" Nov 03 19:18:49.545441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.545358 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv"] Nov 03 19:18:49.545992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.545977 5988 cpu_manager.go:414] "containerMap: removing container" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" Nov 03 19:18:49.546056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.545992 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" Nov 03 19:18:49.546056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.546005 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ed35ebff-edce-48ce-8eb1-fdc2304a1a94" containerName="revoke-keys" Nov 03 19:18:49.546056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.546010 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed35ebff-edce-48ce-8eb1-fdc2304a1a94" containerName="revoke-keys" Nov 03 19:18:49.546056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.546036 5988 cpu_manager.go:414] "containerMap: removing container" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" Nov 03 19:18:49.546056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.546041 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" Nov 03 19:18:49.546216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.546111 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f79162-e784-4c53-af8d-e7524a5cc763" containerName="main" Nov 03 19:18:49.546216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.546120 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed35ebff-edce-48ce-8eb1-fdc2304a1a94" containerName="revoke-keys" Nov 03 19:18:49.546216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.546133 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="856af342-3dc8-4538-a6a0-e779029486a7" containerName="main" Nov 03 19:18:49.550480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.550463 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.586451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.586414 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.586622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.586557 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.586748 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.586719 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxqn\" (UniqueName: \"kubernetes.io/projected/51e5ea48-fa1f-465f-98b4-103fdc69a276-kube-api-access-2jxqn\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.599278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.599237 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv"] Nov 03 19:18:49.647804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.647770 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:18:49.647964 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.647831 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:18:49.688231 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.688195 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.688385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.688251 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.688385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.688338 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxqn\" (UniqueName: \"kubernetes.io/projected/51e5ea48-fa1f-465f-98b4-103fdc69a276-kube-api-access-2jxqn\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.688930 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.688897 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.690590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.690570 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.745777 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.745751 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxqn\" (UniqueName: \"kubernetes.io/projected/51e5ea48-fa1f-465f-98b4-103fdc69a276-kube-api-access-2jxqn\") pod \"maas-api-revoke-sub-388423c4-vkblv\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:49.867139 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:49.867110 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:50.244710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:50.244655 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv"] Nov 03 19:18:50.246835 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:18:50.246806 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e5ea48_fa1f_465f_98b4_103fdc69a276.slice/crio-c2953e82f26d078fa584eb8d8d6ae2fc96f30f62dd257a8705d9fde3ee32e5d4 WatchSource:0}: Error finding container c2953e82f26d078fa584eb8d8d6ae2fc96f30f62dd257a8705d9fde3ee32e5d4: Status 404 returned error can't find the container with id c2953e82f26d078fa584eb8d8d6ae2fc96f30f62dd257a8705d9fde3ee32e5d4 Nov 03 19:18:50.393292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:50.393256 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:50.455216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:50.455184 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" event={"ID":"51e5ea48-fa1f-465f-98b4-103fdc69a276","Type":"ContainerStarted","Data":"c2953e82f26d078fa584eb8d8d6ae2fc96f30f62dd257a8705d9fde3ee32e5d4"} Nov 03 19:18:51.463098 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:51.463065 5988 generic.go:356] "Generic (PLEG): container finished" podID="51e5ea48-fa1f-465f-98b4-103fdc69a276" containerID="81cea0f17da959333b3989ee769ed09e0c101af8005a15ab7033800ec9ca9d3d" exitCode=0 Nov 03 19:18:51.463513 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:51.463149 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" event={"ID":"51e5ea48-fa1f-465f-98b4-103fdc69a276","Type":"ContainerDied","Data":"81cea0f17da959333b3989ee769ed09e0c101af8005a15ab7033800ec9ca9d3d"} Nov 03 19:18:51.878745 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:51.878661 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:52.088700 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.088649 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:52.618235 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.618210 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:52.724226 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.724143 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-xks-ca\") pod \"51e5ea48-fa1f-465f-98b4-103fdc69a276\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " Nov 03 19:18:52.724388 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.724232 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-openshift-ca\") pod \"51e5ea48-fa1f-465f-98b4-103fdc69a276\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " Nov 03 19:18:52.724388 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.724359 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e5ea48-fa1f-465f-98b4-103fdc69a276-kube-api-access-2jxqn\" (UniqueName: \"kubernetes.io/projected/51e5ea48-fa1f-465f-98b4-103fdc69a276-kube-api-access-2jxqn\") pod \"51e5ea48-fa1f-465f-98b4-103fdc69a276\" (UID: \"51e5ea48-fa1f-465f-98b4-103fdc69a276\") " Nov 03 19:18:52.724716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.724663 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-openshift-ca" pod "51e5ea48-fa1f-465f-98b4-103fdc69a276" (UID: "51e5ea48-fa1f-465f-98b4-103fdc69a276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:18:52.724911 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.724895 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:52.726251 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.726225 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-xks-ca" pod "51e5ea48-fa1f-465f-98b4-103fdc69a276" (UID: "51e5ea48-fa1f-465f-98b4-103fdc69a276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:18:52.726469 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.726448 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e5ea48-fa1f-465f-98b4-103fdc69a276-kube-api-access-2jxqn" pod "51e5ea48-fa1f-465f-98b4-103fdc69a276" (UID: "51e5ea48-fa1f-465f-98b4-103fdc69a276"). InnerVolumeSpecName "kube-api-access-2jxqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:18:52.826036 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.825998 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e5ea48-fa1f-465f-98b4-103fdc69a276-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:52.826036 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:52.826032 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jxqn\" (UniqueName: \"kubernetes.io/projected/51e5ea48-fa1f-465f-98b4-103fdc69a276-kube-api-access-2jxqn\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:18:53.477380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:53.477339 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" event={"ID":"51e5ea48-fa1f-465f-98b4-103fdc69a276","Type":"ContainerDied","Data":"c2953e82f26d078fa584eb8d8d6ae2fc96f30f62dd257a8705d9fde3ee32e5d4"} Nov 03 19:18:53.477380 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:53.477377 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv" Nov 03 19:18:53.477591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:53.477401 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2953e82f26d078fa584eb8d8d6ae2fc96f30f62dd257a8705d9fde3ee32e5d4" Nov 03 19:18:54.878333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:54.878290 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:55.322741 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:55.322698 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:55.535037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:55.535000 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:55.740295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:18:55.740177 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:18:55.937758 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:18:55.937713 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:19:00.120656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:00.120609 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:03.171353 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:03.171304 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:03.385432 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:03.385393 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:04.205552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:04.205513 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:04.847477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:04.847430 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:08.272289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:08.272242 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:08.477713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:08.477656 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:08.723738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:08.716846 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr"] Nov 03 19:19:08.723738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:08.717101 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" podUID="a6e29b9f-c2a3-464e-9e89-22e3aefad77c" containerName="maas-api" containerID="cri-o://a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709" gracePeriod=30 Nov 03 19:19:08.868667 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:08.868624 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:09.048540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.048509 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:19:09.066516 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.066482 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls\") pod \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " Nov 03 19:19:09.066702 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.066639 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-openshift-service-ca\") pod \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " Nov 03 19:19:09.066702 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.066697 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-kube-api-access-s29z2\" (UniqueName: \"kubernetes.io/projected/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-kube-api-access-s29z2\") pod \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " Nov 03 19:19:09.066876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.066846 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-metrics-certs\") pod \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\" (UID: \"a6e29b9f-c2a3-464e-9e89-22e3aefad77c\") " Nov 03 19:19:09.067092 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.067058 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-openshift-service-ca" pod "a6e29b9f-c2a3-464e-9e89-22e3aefad77c" (UID: "a6e29b9f-c2a3-464e-9e89-22e3aefad77c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:09.067875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.067847 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:09.068806 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.068781 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls" pod "a6e29b9f-c2a3-464e-9e89-22e3aefad77c" (UID: "a6e29b9f-c2a3-464e-9e89-22e3aefad77c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:09.069076 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.069055 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-kube-api-access-s29z2" pod "a6e29b9f-c2a3-464e-9e89-22e3aefad77c" (UID: "a6e29b9f-c2a3-464e-9e89-22e3aefad77c"). InnerVolumeSpecName "kube-api-access-s29z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:09.069356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.069332 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-metrics-certs" pod "a6e29b9f-c2a3-464e-9e89-22e3aefad77c" (UID: "a6e29b9f-c2a3-464e-9e89-22e3aefad77c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:09.169302 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.169258 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s29z2\" (UniqueName: \"kubernetes.io/projected/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-kube-api-access-s29z2\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:09.169302 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.169303 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:09.169556 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.169317 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e29b9f-c2a3-464e-9e89-22e3aefad77c-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:09.601293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.601257 5988 generic.go:356] "Generic (PLEG): container finished" podID="a6e29b9f-c2a3-464e-9e89-22e3aefad77c" containerID="a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709" exitCode=0 Nov 03 19:19:09.601843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.601332 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" Nov 03 19:19:09.601843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.601344 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" event={"ID":"a6e29b9f-c2a3-464e-9e89-22e3aefad77c","Type":"ContainerDied","Data":"a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709"} Nov 03 19:19:09.601843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.601419 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr" event={"ID":"a6e29b9f-c2a3-464e-9e89-22e3aefad77c","Type":"ContainerDied","Data":"aed3986c9fb3517c9353fae39fa8eea4144acde99562da776df14c8d9fce2fac"} Nov 03 19:19:09.601843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.601452 5988 scope.go:122] "RemoveContainer" containerID="a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709" Nov 03 19:19:09.613546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.613521 5988 scope.go:122] "RemoveContainer" containerID="a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709" Nov 03 19:19:09.613961 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:19:09.613936 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709\": container with ID starting with a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709 not found: ID does not exist" containerID="a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709" Nov 03 19:19:09.614046 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.613973 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709"} err="failed to get container status \"a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709\": rpc error: code = NotFound desc = could not find container \"a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709\": container with ID starting with a24475c7789827d220cbe278357405ef9b8efe27540537b4c627b2ef9a692709 not found: ID does not exist" Nov 03 19:19:09.760745 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.760707 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr"] Nov 03 19:19:09.767158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:09.767126 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-69e2b9-5657568589-r8nxr"] Nov 03 19:19:10.382029 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:10.381985 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:10.588894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:10.588860 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:10.798203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:10.798116 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:11.008990 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:11.008941 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:11.273833 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:11.273801 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e29b9f-c2a3-464e-9e89-22e3aefad77c" path="/var/lib/kubelet/pods/a6e29b9f-c2a3-464e-9e89-22e3aefad77c/volumes" Nov 03 19:19:11.420117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:11.420084 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:12.880409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:12.880344 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:15.007145 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.007105 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m"] Nov 03 19:19:15.007931 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.007909 5988 cpu_manager.go:414] "containerMap: removing container" podUID="51e5ea48-fa1f-465f-98b4-103fdc69a276" containerName="revoke-keys" Nov 03 19:19:15.008045 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.007933 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e5ea48-fa1f-465f-98b4-103fdc69a276" containerName="revoke-keys" Nov 03 19:19:15.008045 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.007990 5988 cpu_manager.go:414] "containerMap: removing container" podUID="a6e29b9f-c2a3-464e-9e89-22e3aefad77c" containerName="maas-api" Nov 03 19:19:15.008045 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.007999 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e29b9f-c2a3-464e-9e89-22e3aefad77c" containerName="maas-api" Nov 03 19:19:15.008222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.008140 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e5ea48-fa1f-465f-98b4-103fdc69a276" containerName="revoke-keys" Nov 03 19:19:15.008222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.008165 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e29b9f-c2a3-464e-9e89-22e3aefad77c" containerName="maas-api" Nov 03 19:19:15.012791 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.012767 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.043240 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.043206 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.043484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.043459 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.043547 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.043508 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4rm\" (UniqueName: \"kubernetes.io/projected/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-kube-api-access-xj4rm\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.059561 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.059528 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl"] Nov 03 19:19:15.063732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.063707 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.065937 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.065890 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m"] Nov 03 19:19:15.103614 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.103585 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:19:15.103834 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.103643 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:19:15.118639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.118605 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl"] Nov 03 19:19:15.145280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.145247 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.145448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.145304 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.145448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.145395 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2wx\" (UniqueName: \"kubernetes.io/projected/a6c2e453-db67-4b49-9474-03272e83175b-kube-api-access-nk2wx\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.145574 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.145548 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.145634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.145590 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.145724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.145657 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4rm\" (UniqueName: \"kubernetes.io/projected/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-kube-api-access-xj4rm\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.146216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.146193 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.147657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.147638 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.203470 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.203434 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4rm\" (UniqueName: \"kubernetes.io/projected/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-kube-api-access-xj4rm\") pod \"maas-api-revoke-sub-d117c998-zvr5m\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.247470 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.247437 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2wx\" (UniqueName: \"kubernetes.io/projected/a6c2e453-db67-4b49-9474-03272e83175b-kube-api-access-nk2wx\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.247665 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.247484 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.247665 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.247571 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.248212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.248184 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.249877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.249859 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.304916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.304852 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2wx\" (UniqueName: \"kubernetes.io/projected/a6c2e453-db67-4b49-9474-03272e83175b-kube-api-access-nk2wx\") pod \"maas-api-revoke-sub-b1bf6db0-gvddl\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.327227 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.327193 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:15.380117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.380085 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:15.698403 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.698376 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m"] Nov 03 19:19:15.700342 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:19:15.700303 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf178a2bb_b5b0_43d5_b6e2_85d5fa9bdbb7.slice/crio-484b9d7cf6083a62fff3ba41d6744c932e20d37961e15916c459cd9201db2230 WatchSource:0}: Error finding container 484b9d7cf6083a62fff3ba41d6744c932e20d37961e15916c459cd9201db2230: Status 404 returned error can't find the container with id 484b9d7cf6083a62fff3ba41d6744c932e20d37961e15916c459cd9201db2230 Nov 03 19:19:15.803128 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.803096 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl"] Nov 03 19:19:15.804932 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:19:15.804905 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c2e453_db67_4b49_9474_03272e83175b.slice/crio-8aedad052fc8385f88ba10ae21ff329f645b4db66d47053f51b27a727a196d5d WatchSource:0}: Error finding container 8aedad052fc8385f88ba10ae21ff329f645b4db66d47053f51b27a727a196d5d: Status 404 returned error can't find the container with id 8aedad052fc8385f88ba10ae21ff329f645b4db66d47053f51b27a727a196d5d Nov 03 19:19:15.946585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:15.946508 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:16.429283 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:16.429245 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:16.649405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:16.649373 5988 generic.go:356] "Generic (PLEG): container finished" podID="f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" containerID="cd35e4e8c04a21176cb14b0203e61f8decbea833e022f8dd6cba8d25236baa71" exitCode=0 Nov 03 19:19:16.649594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:16.649502 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" event={"ID":"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7","Type":"ContainerDied","Data":"cd35e4e8c04a21176cb14b0203e61f8decbea833e022f8dd6cba8d25236baa71"} Nov 03 19:19:16.649594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:16.649582 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" event={"ID":"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7","Type":"ContainerStarted","Data":"484b9d7cf6083a62fff3ba41d6744c932e20d37961e15916c459cd9201db2230"} Nov 03 19:19:16.651434 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:16.651396 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" event={"ID":"a6c2e453-db67-4b49-9474-03272e83175b","Type":"ContainerStarted","Data":"a32e185513b11a3f41a35aff852e448050c121100295ac5a2a678e411b883332"} Nov 03 19:19:16.651546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:16.651453 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" event={"ID":"a6c2e453-db67-4b49-9474-03272e83175b","Type":"ContainerStarted","Data":"8aedad052fc8385f88ba10ae21ff329f645b4db66d47053f51b27a727a196d5d"} Nov 03 19:19:17.659925 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.659889 5988 generic.go:356] "Generic (PLEG): container finished" podID="a6c2e453-db67-4b49-9474-03272e83175b" containerID="a32e185513b11a3f41a35aff852e448050c121100295ac5a2a678e411b883332" exitCode=0 Nov 03 19:19:17.660434 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.659976 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" event={"ID":"a6c2e453-db67-4b49-9474-03272e83175b","Type":"ContainerDied","Data":"a32e185513b11a3f41a35aff852e448050c121100295ac5a2a678e411b883332"} Nov 03 19:19:17.822603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.822578 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:17.859100 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.859068 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:17.880662 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.880634 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-kube-api-access-xj4rm\" (UniqueName: \"kubernetes.io/projected/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-kube-api-access-xj4rm\") pod \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " Nov 03 19:19:17.880826 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.880672 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-openshift-ca\") pod \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " Nov 03 19:19:17.880826 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.880740 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-xks-ca\") pod \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\" (UID: \"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7\") " Nov 03 19:19:17.881063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.881037 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-openshift-ca" pod "f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" (UID: "f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:17.881183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.881168 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:17.882718 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.882698 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-kube-api-access-xj4rm" pod "f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" (UID: "f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7"). InnerVolumeSpecName "kube-api-access-xj4rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:17.882801 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.882714 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-xks-ca" pod "f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" (UID: "f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:17.982898 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.982823 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj4rm\" (UniqueName: \"kubernetes.io/projected/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-kube-api-access-xj4rm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:17.982898 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:17.982852 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:18.073147 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.073116 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:18.106722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.106662 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx"] Nov 03 19:19:18.106910 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.106888 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" containerID="cri-o://6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a" gracePeriod=30 Nov 03 19:19:18.351971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.351937 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:18.668426 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.668358 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" Nov 03 19:19:18.668426 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.668408 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m" event={"ID":"f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7","Type":"ContainerDied","Data":"484b9d7cf6083a62fff3ba41d6744c932e20d37961e15916c459cd9201db2230"} Nov 03 19:19:18.668877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.668462 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="484b9d7cf6083a62fff3ba41d6744c932e20d37961e15916c459cd9201db2230" Nov 03 19:19:18.808724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.808673 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:18.893728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.893701 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-xks-ca\") pod \"a6c2e453-db67-4b49-9474-03272e83175b\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " Nov 03 19:19:18.893876 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.893799 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c2e453-db67-4b49-9474-03272e83175b-kube-api-access-nk2wx\" (UniqueName: \"kubernetes.io/projected/a6c2e453-db67-4b49-9474-03272e83175b-kube-api-access-nk2wx\") pod \"a6c2e453-db67-4b49-9474-03272e83175b\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " Nov 03 19:19:18.893955 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.893943 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-openshift-ca\") pod \"a6c2e453-db67-4b49-9474-03272e83175b\" (UID: \"a6c2e453-db67-4b49-9474-03272e83175b\") " Nov 03 19:19:18.894282 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.894259 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-openshift-ca" pod "a6c2e453-db67-4b49-9474-03272e83175b" (UID: "a6c2e453-db67-4b49-9474-03272e83175b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:18.895886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.895862 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c2e453-db67-4b49-9474-03272e83175b-kube-api-access-nk2wx" pod "a6c2e453-db67-4b49-9474-03272e83175b" (UID: "a6c2e453-db67-4b49-9474-03272e83175b"). InnerVolumeSpecName "kube-api-access-nk2wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:18.895886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.895870 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-xks-ca" pod "a6c2e453-db67-4b49-9474-03272e83175b" (UID: "a6c2e453-db67-4b49-9474-03272e83175b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:18.994945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.994865 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:18.994945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.994893 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk2wx\" (UniqueName: \"kubernetes.io/projected/a6c2e453-db67-4b49-9474-03272e83175b-kube-api-access-nk2wx\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:18.994945 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:18.994902 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6c2e453-db67-4b49-9474-03272e83175b-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:19.016615 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.016584 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q"] Nov 03 19:19:19.019885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.018056 5988 cpu_manager.go:414] "containerMap: removing container" podUID="a6c2e453-db67-4b49-9474-03272e83175b" containerName="revoke-keys" Nov 03 19:19:19.019885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.018081 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c2e453-db67-4b49-9474-03272e83175b" containerName="revoke-keys" Nov 03 19:19:19.019885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.018173 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" containerName="revoke-keys" Nov 03 19:19:19.019885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.018182 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" containerName="revoke-keys" Nov 03 19:19:19.019885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.018468 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" containerName="revoke-keys" Nov 03 19:19:19.019885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.018500 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c2e453-db67-4b49-9474-03272e83175b" containerName="revoke-keys" Nov 03 19:19:19.024232 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.024207 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.073282 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.073252 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q"] Nov 03 19:19:19.096540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.096509 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm7tq\" (UniqueName: \"kubernetes.io/projected/7edbf57b-2dae-4e0f-8682-b08504feca4f-kube-api-access-xm7tq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.096711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.096668 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.096813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.096790 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.197614 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.197577 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm7tq\" (UniqueName: \"kubernetes.io/projected/7edbf57b-2dae-4e0f-8682-b08504feca4f-kube-api-access-xm7tq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.197837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.197670 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.197837 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.197758 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.198430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.198409 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.200130 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.200107 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.254935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.254881 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm7tq\" (UniqueName: \"kubernetes.io/projected/7edbf57b-2dae-4e0f-8682-b08504feca4f-kube-api-access-xm7tq\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.344977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.344952 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:19.676076 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.676043 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" Nov 03 19:19:19.676447 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.676070 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl" event={"ID":"a6c2e453-db67-4b49-9474-03272e83175b","Type":"ContainerDied","Data":"8aedad052fc8385f88ba10ae21ff329f645b4db66d47053f51b27a727a196d5d"} Nov 03 19:19:19.676447 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.676114 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aedad052fc8385f88ba10ae21ff329f645b4db66d47053f51b27a727a196d5d" Nov 03 19:19:19.712978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:19.712953 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q"] Nov 03 19:19:19.714532 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:19:19.714496 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edbf57b_2dae_4e0f_8682_b08504feca4f.slice/crio-fee72dce4ce7765eeb8c5ac1c0ecc75f3dbb4af432ce18fe56c005ad0b0ab3cc WatchSource:0}: Error finding container fee72dce4ce7765eeb8c5ac1c0ecc75f3dbb4af432ce18fe56c005ad0b0ab3cc: Status 404 returned error can't find the container with id fee72dce4ce7765eeb8c5ac1c0ecc75f3dbb4af432ce18fe56c005ad0b0ab3cc Nov 03 19:19:20.683841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:20.683806 5988 generic.go:356] "Generic (PLEG): container finished" podID="7edbf57b-2dae-4e0f-8682-b08504feca4f" containerID="47b1dfb14909a796012f317015e7046c97a094716cc39806d3c3c1a112686e78" exitCode=0 Nov 03 19:19:20.684248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:20.683892 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" event={"ID":"7edbf57b-2dae-4e0f-8682-b08504feca4f","Type":"ContainerDied","Data":"47b1dfb14909a796012f317015e7046c97a094716cc39806d3c3c1a112686e78"} Nov 03 19:19:20.684248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:20.683946 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" event={"ID":"7edbf57b-2dae-4e0f-8682-b08504feca4f","Type":"ContainerStarted","Data":"fee72dce4ce7765eeb8c5ac1c0ecc75f3dbb4af432ce18fe56c005ad0b0ab3cc"} Nov 03 19:19:20.778592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:20.778567 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:19:20.904377 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:20.904339 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:21.843298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.843273 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:21.928579 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.928537 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-xks-ca\") pod \"7edbf57b-2dae-4e0f-8682-b08504feca4f\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " Nov 03 19:19:21.928819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.928602 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7edbf57b-2dae-4e0f-8682-b08504feca4f-kube-api-access-xm7tq\" (UniqueName: \"kubernetes.io/projected/7edbf57b-2dae-4e0f-8682-b08504feca4f-kube-api-access-xm7tq\") pod \"7edbf57b-2dae-4e0f-8682-b08504feca4f\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " Nov 03 19:19:21.928819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.928711 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-openshift-ca\") pod \"7edbf57b-2dae-4e0f-8682-b08504feca4f\" (UID: \"7edbf57b-2dae-4e0f-8682-b08504feca4f\") " Nov 03 19:19:21.929043 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.929014 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-openshift-ca" pod "7edbf57b-2dae-4e0f-8682-b08504feca4f" (UID: "7edbf57b-2dae-4e0f-8682-b08504feca4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:21.929214 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.929198 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:21.930783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.930752 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7edbf57b-2dae-4e0f-8682-b08504feca4f-kube-api-access-xm7tq" pod "7edbf57b-2dae-4e0f-8682-b08504feca4f" (UID: "7edbf57b-2dae-4e0f-8682-b08504feca4f"). InnerVolumeSpecName "kube-api-access-xm7tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:21.930783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:21.930758 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-xks-ca" pod "7edbf57b-2dae-4e0f-8682-b08504feca4f" (UID: "7edbf57b-2dae-4e0f-8682-b08504feca4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:22.030138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:22.030054 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7edbf57b-2dae-4e0f-8682-b08504feca4f-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:22.030138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:22.030086 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm7tq\" (UniqueName: \"kubernetes.io/projected/7edbf57b-2dae-4e0f-8682-b08504feca4f-kube-api-access-xm7tq\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:22.699694 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:22.699657 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" Nov 03 19:19:22.699906 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:22.699782 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q" event={"ID":"7edbf57b-2dae-4e0f-8682-b08504feca4f","Type":"ContainerDied","Data":"fee72dce4ce7765eeb8c5ac1c0ecc75f3dbb4af432ce18fe56c005ad0b0ab3cc"} Nov 03 19:19:22.699906 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:22.699824 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee72dce4ce7765eeb8c5ac1c0ecc75f3dbb4af432ce18fe56c005ad0b0ab3cc" Nov 03 19:19:23.479312 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.479282 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:19:23.587616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.587580 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-data\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.587843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.587710 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-certs\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.587843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.587742 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83803762-e846-4658-812c-a5dc412ebb1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83803762-e846-4658-812c-a5dc412ebb1f-istio-podinfo\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.587843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.587774 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-istio-token\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-istio-token\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.588001 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.587849 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-socket\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.588001 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.587889 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-credential-socket\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.588001 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.587946 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-data" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:23.588163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588015 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83803762-e846-4658-812c-a5dc412ebb1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83803762-e846-4658-812c-a5dc412ebb1f-istiod-ca-cert\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.588163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588106 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-envoy\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.588163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588153 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-kube-api-access-7jzrk\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-kube-api-access-7jzrk\") pod \"83803762-e846-4658-812c-a5dc412ebb1f\" (UID: \"83803762-e846-4658-812c-a5dc412ebb1f\") " Nov 03 19:19:23.588327 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588259 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-certs" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:23.588419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588392 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83803762-e846-4658-812c-a5dc412ebb1f-istiod-ca-cert" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:23.588553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588526 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-socket" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:23.588695 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588550 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-credential-socket" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:23.588695 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588670 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83803762-e846-4658-812c-a5dc412ebb1f-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.588812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588709 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.588812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.588723 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.590400 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.590374 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-istio-token" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:23.590605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.590579 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-kube-api-access-7jzrk" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "kube-api-access-7jzrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:23.590714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.590641 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-envoy" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:19:23.590714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.590647 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83803762-e846-4658-812c-a5dc412ebb1f-istio-podinfo" pod "83803762-e846-4658-812c-a5dc412ebb1f" (UID: "83803762-e846-4658-812c-a5dc412ebb1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:19:23.689994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.689911 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.689994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.689946 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzrk\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-kube-api-access-7jzrk\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.689994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.689957 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83803762-e846-4658-812c-a5dc412ebb1f-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.689994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.689965 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83803762-e846-4658-812c-a5dc412ebb1f-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.689994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.689974 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.689994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.689983 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83803762-e846-4658-812c-a5dc412ebb1f-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:23.708015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.707982 5988 generic.go:356] "Generic (PLEG): container finished" podID="83803762-e846-4658-812c-a5dc412ebb1f" containerID="6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a" exitCode=0 Nov 03 19:19:23.708175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.708036 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" Nov 03 19:19:23.708175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.708075 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" event={"ID":"83803762-e846-4658-812c-a5dc412ebb1f","Type":"ContainerDied","Data":"6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a"} Nov 03 19:19:23.708175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.708137 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx" event={"ID":"83803762-e846-4658-812c-a5dc412ebb1f","Type":"ContainerDied","Data":"6b0a33eb10c55e33782be1c31b343fa3711d72c566fabd8c817a56093d8f6aad"} Nov 03 19:19:23.708175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.708167 5988 scope.go:122] "RemoveContainer" containerID="6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a" Nov 03 19:19:23.719734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.719711 5988 scope.go:122] "RemoveContainer" containerID="0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f" Nov 03 19:19:23.729579 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.729559 5988 scope.go:122] "RemoveContainer" containerID="6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a" Nov 03 19:19:23.729895 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:19:23.729870 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a\": container with ID starting with 6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a not found: ID does not exist" containerID="6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a" Nov 03 19:19:23.729962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.729905 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a"} err="failed to get container status \"6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a\": rpc error: code = NotFound desc = could not find container \"6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a\": container with ID starting with 6e262bc8cf80a9641c6f2b677f2f72a95172110b5a2f9a256e33b35b3f86ec5a not found: ID does not exist" Nov 03 19:19:23.729962 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.729924 5988 scope.go:122] "RemoveContainer" containerID="0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f" Nov 03 19:19:23.730210 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:19:23.730178 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f\": container with ID starting with 0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f not found: ID does not exist" containerID="0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f" Nov 03 19:19:23.730267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.730215 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f"} err="failed to get container status \"0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f\": rpc error: code = NotFound desc = could not find container \"0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f\": container with ID starting with 0137b985a215f81d01d8a68bb2553e3004564f50ecf7f2bc981a8cbd3f9e9b4f not found: ID does not exist" Nov 03 19:19:23.872651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.872564 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx"] Nov 03 19:19:23.878102 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:23.878073 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-69e2b9-openshift-default-545d9ffd69-nhgvx"] Nov 03 19:19:25.077092 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:25.076967 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:25.287071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:25.287036 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83803762-e846-4658-812c-a5dc412ebb1f" path="/var/lib/kubelet/pods/83803762-e846-4658-812c-a5dc412ebb1f/volumes" Nov 03 19:19:25.936075 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:19:25.936037 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:19:26.605850 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:26.605778 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:26.810594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:26.810559 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:27.042518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:27.042411 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:27.055337 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:27.055308 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7"] Nov 03 19:19:27.095067 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:27.095036 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-3a7056-g84n7"] Nov 03 19:19:27.276242 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:27.276209 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51a992e-7a43-4b7d-9765-5a1741254d80" path="/var/lib/kubelet/pods/c51a992e-7a43-4b7d-9765-5a1741254d80/volumes" Nov 03 19:19:27.482234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:27.482205 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:30.560866 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:30.560829 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq"] Nov 03 19:19:30.674105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:30.674073 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm"] Nov 03 19:19:31.002927 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:31.002803 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:31.214494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:31.214460 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:31.614498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:31.614460 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:31.827122 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:31.827089 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:31.855564 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:31.855534 5988 patch_prober.go:29] interesting pod/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": read tcp 10.134.0.2:59216->10.134.0.112:15021: read: connection reset by peer" start-of-body= Nov 03 19:19:31.855749 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:31.855611 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": read tcp 10.134.0.2:59216->10.134.0.112:15021: read: connection reset by peer" Nov 03 19:19:32.768570 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:32.768536 5988 generic.go:356] "Generic (PLEG): container finished" podID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerID="1efeb9a34e39b9fc08ff33b8092f496221d04043f596a483c21091f06902a992" exitCode=0 Nov 03 19:19:32.768986 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:32.768606 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" event={"ID":"b67e964c-219b-43c9-9e23-0e1aa854fbff","Type":"ContainerDied","Data":"1efeb9a34e39b9fc08ff33b8092f496221d04043f596a483c21091f06902a992"} Nov 03 19:19:32.769051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:32.769028 5988 scope.go:122] "RemoveContainer" containerID="1efeb9a34e39b9fc08ff33b8092f496221d04043f596a483c21091f06902a992" Nov 03 19:19:32.773561 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:32.773527 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:19:32.773710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:32.773613 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:19:32.773710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:32.773660 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:19:33.779753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:33.779717 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" event={"ID":"b67e964c-219b-43c9-9e23-0e1aa854fbff","Type":"ContainerStarted","Data":"b97b1be2735ded6d25b6a60eee4933f865ef1ab34bc7bf2c23bf70130a66daa4"} Nov 03 19:19:33.882266 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:33.882232 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:34.095574 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:34.095497 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:34.130022 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:34.129993 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:19:34.130212 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:34.130185 5988 patch_prober.go:29] interesting pod/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:19:34.130291 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:34.130246 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:19:34.303362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:34.303324 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:35.130632 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.130599 5988 patch_prober.go:29] interesting pod/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:19:35.131061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.130647 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:19:35.565013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.564746 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" podUID="3beb55de-3f42-4add-98e2-123ecfe00075" containerName="payload-processing" containerID="cri-o://56b6af758cde5cce7c95ded6329998e3707aa6f67196cba6f91a9dfa7633fc4d" gracePeriod=25 Nov 03 19:19:35.675842 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.675580 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" podUID="81c94319-5dd5-4e55-ae3f-f5f96e3f2778" containerName="payload-pre-processing" containerID="cri-o://83ca87c52bd63bd1014d992ad3c61b13092b11f9de25da85fe3e3fed0af38742" gracePeriod=25 Nov 03 19:19:35.687218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.687177 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:35.819997 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.819964 5988 generic.go:356] "Generic (PLEG): container finished" podID="81c94319-5dd5-4e55-ae3f-f5f96e3f2778" containerID="83ca87c52bd63bd1014d992ad3c61b13092b11f9de25da85fe3e3fed0af38742" exitCode=0 Nov 03 19:19:35.820276 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.820221 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" event={"ID":"81c94319-5dd5-4e55-ae3f-f5f96e3f2778","Type":"ContainerDied","Data":"83ca87c52bd63bd1014d992ad3c61b13092b11f9de25da85fe3e3fed0af38742"} Nov 03 19:19:35.822171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.822099 5988 generic.go:356] "Generic (PLEG): container finished" podID="3beb55de-3f42-4add-98e2-123ecfe00075" containerID="56b6af758cde5cce7c95ded6329998e3707aa6f67196cba6f91a9dfa7633fc4d" exitCode=0 Nov 03 19:19:35.822271 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.822200 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" event={"ID":"3beb55de-3f42-4add-98e2-123ecfe00075","Type":"ContainerDied","Data":"56b6af758cde5cce7c95ded6329998e3707aa6f67196cba6f91a9dfa7633fc4d"} Nov 03 19:19:35.976281 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:35.975964 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:19:36.130221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.130129 5988 patch_prober.go:29] interesting pod/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:19:36.130221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.130208 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:19:36.136123 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.134427 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beb55de-3f42-4add-98e2-123ecfe00075-kube-api-access-shtlq\" (UniqueName: \"kubernetes.io/projected/3beb55de-3f42-4add-98e2-123ecfe00075-kube-api-access-shtlq\") pod \"3beb55de-3f42-4add-98e2-123ecfe00075\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " Nov 03 19:19:36.136123 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.134618 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3beb55de-3f42-4add-98e2-123ecfe00075-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beb55de-3f42-4add-98e2-123ecfe00075-plugins-config-volume\") pod \"3beb55de-3f42-4add-98e2-123ecfe00075\" (UID: \"3beb55de-3f42-4add-98e2-123ecfe00075\") " Nov 03 19:19:36.136123 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.135186 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb55de-3f42-4add-98e2-123ecfe00075-plugins-config-volume" pod "3beb55de-3f42-4add-98e2-123ecfe00075" (UID: "3beb55de-3f42-4add-98e2-123ecfe00075"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:36.136123 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.135277 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3beb55de-3f42-4add-98e2-123ecfe00075-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:36.137022 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.136995 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beb55de-3f42-4add-98e2-123ecfe00075-kube-api-access-shtlq" pod "3beb55de-3f42-4add-98e2-123ecfe00075" (UID: "3beb55de-3f42-4add-98e2-123ecfe00075"). InnerVolumeSpecName "kube-api-access-shtlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:36.183719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.177595 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:36.185712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.185429 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:19:36.235736 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.235705 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-kube-api-access-7qckq\" (UniqueName: \"kubernetes.io/projected/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-kube-api-access-7qckq\") pod \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " Nov 03 19:19:36.235940 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.235758 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-plugins-config-volume\") pod \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\" (UID: \"81c94319-5dd5-4e55-ae3f-f5f96e3f2778\") " Nov 03 19:19:36.236225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.236200 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-plugins-config-volume" pod "81c94319-5dd5-4e55-ae3f-f5f96e3f2778" (UID: "81c94319-5dd5-4e55-ae3f-f5f96e3f2778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:36.236225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.236217 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtlq\" (UniqueName: \"kubernetes.io/projected/3beb55de-3f42-4add-98e2-123ecfe00075-kube-api-access-shtlq\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:36.237788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.237765 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-kube-api-access-7qckq" pod "81c94319-5dd5-4e55-ae3f-f5f96e3f2778" (UID: "81c94319-5dd5-4e55-ae3f-f5f96e3f2778"). InnerVolumeSpecName "kube-api-access-7qckq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:36.338181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.338143 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qckq\" (UniqueName: \"kubernetes.io/projected/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-kube-api-access-7qckq\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:36.338181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.338182 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81c94319-5dd5-4e55-ae3f-f5f96e3f2778-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:36.377435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.377401 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:36.832757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.832718 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" event={"ID":"81c94319-5dd5-4e55-ae3f-f5f96e3f2778","Type":"ContainerDied","Data":"bc354f4b26d7f4082e23e4d6954b9f213f09ced77440d638aac70f84bee630c7"} Nov 03 19:19:36.832757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.832753 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm" Nov 03 19:19:36.833018 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.832795 5988 scope.go:122] "RemoveContainer" containerID="83ca87c52bd63bd1014d992ad3c61b13092b11f9de25da85fe3e3fed0af38742" Nov 03 19:19:36.834725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.834592 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" Nov 03 19:19:36.834725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.834625 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq" event={"ID":"3beb55de-3f42-4add-98e2-123ecfe00075","Type":"ContainerDied","Data":"0300a7f164c1e73f81b15ccfc8be642e0bb6fd1724e89cd8ecc02680166a0700"} Nov 03 19:19:36.848624 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.846018 5988 scope.go:122] "RemoveContainer" containerID="56b6af758cde5cce7c95ded6329998e3707aa6f67196cba6f91a9dfa7633fc4d" Nov 03 19:19:36.988135 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.988098 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq"] Nov 03 19:19:36.991458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:36.991431 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9e277a-5dc7dd98bc-dt7fq"] Nov 03 19:19:37.130284 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.130205 5988 patch_prober.go:29] interesting pod/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:19:37.130284 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.130251 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:19:37.144797 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.136238 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm"] Nov 03 19:19:37.145506 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.145477 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9e277a-6f7b5d4955-p29sm"] Nov 03 19:19:37.178494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.178458 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:37.279457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.279427 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3beb55de-3f42-4add-98e2-123ecfe00075" path="/var/lib/kubelet/pods/3beb55de-3f42-4add-98e2-123ecfe00075/volumes" Nov 03 19:19:37.279923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.279903 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c94319-5dd5-4e55-ae3f-f5f96e3f2778" path="/var/lib/kubelet/pods/81c94319-5dd5-4e55-ae3f-f5f96e3f2778/volumes" Nov 03 19:19:37.377429 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.377398 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:37.616370 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.616328 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q"] Nov 03 19:19:37.620322 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.620290 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9e277a-h5d8q"] Nov 03 19:19:37.659706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:37.658189 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:38.078497 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.078464 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:38.129769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.129732 5988 patch_prober.go:29] interesting pod/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 19:19:38.129965 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.129793 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 19:19:38.271712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.270952 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl"] Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271831 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3beb55de-3f42-4add-98e2-123ecfe00075" containerName="payload-processing" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271853 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3beb55de-3f42-4add-98e2-123ecfe00075" containerName="payload-processing" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271889 5988 cpu_manager.go:414] "containerMap: removing container" podUID="81c94319-5dd5-4e55-ae3f-f5f96e3f2778" containerName="payload-pre-processing" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271899 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c94319-5dd5-4e55-ae3f-f5f96e3f2778" containerName="payload-pre-processing" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271931 5988 cpu_manager.go:414] "containerMap: removing container" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271938 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271955 5988 cpu_manager.go:414] "containerMap: removing container" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271962 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271978 5988 cpu_manager.go:414] "containerMap: removing container" podUID="7edbf57b-2dae-4e0f-8682-b08504feca4f" containerName="revoke-keys" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.271986 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7edbf57b-2dae-4e0f-8682-b08504feca4f" containerName="revoke-keys" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.272129 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3beb55de-3f42-4add-98e2-123ecfe00075" containerName="payload-processing" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.272143 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c94319-5dd5-4e55-ae3f-f5f96e3f2778" containerName="payload-pre-processing" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.272156 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.272168 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="83803762-e846-4658-812c-a5dc412ebb1f" containerName="istio-proxy" Nov 03 19:19:38.272233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.272179 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="7edbf57b-2dae-4e0f-8682-b08504feca4f" containerName="revoke-keys" Nov 03 19:19:38.283703 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.282287 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.291451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.291410 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:38.318291 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.318262 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl"] Nov 03 19:19:38.331903 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.331822 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d74f801\"/\"openshift-service-ca.crt\"" Nov 03 19:19:38.370995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.370963 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d74f801\"/\"kube-root-ca.crt\"" Nov 03 19:19:38.378053 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.378023 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-dshm\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.378201 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.378084 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-home\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.378201 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.378144 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-model-cache\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.378327 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.378264 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858e40ab-0455-4e17-bc4d-ff6163b3e750-tls-certs\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.378375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.378361 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhxn\" (UniqueName: \"kubernetes.io/projected/858e40ab-0455-4e17-bc4d-ff6163b3e750-kube-api-access-lfhxn\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.378439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.378422 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-tmp-dir\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.400180 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.400123 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d74f801\"/\"e2e-lifecycle-model-3d74f801-kserve-self-signed-certs\"" Nov 03 19:19:38.400400 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.400277 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d74f801\"/\"default-dockercfg-n6v2w\"" Nov 03 19:19:38.479153 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479118 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-dshm\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479163 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-home\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479181 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-model-cache\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479235 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858e40ab-0455-4e17-bc4d-ff6163b3e750-tls-certs\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479304 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhxn\" (UniqueName: \"kubernetes.io/projected/858e40ab-0455-4e17-bc4d-ff6163b3e750-kube-api-access-lfhxn\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479362 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-tmp-dir\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479677 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-model-cache\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479852 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479748 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-tmp-dir\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.479977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.479951 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-home\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.481637 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.481613 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-dshm\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.481985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.481966 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858e40ab-0455-4e17-bc4d-ff6163b3e750-tls-certs\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.499116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.499093 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:38.536889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.536863 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhxn\" (UniqueName: \"kubernetes.io/projected/858e40ab-0455-4e17-bc4d-ff6163b3e750-kube-api-access-lfhxn\") pod \"e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:38.600107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:38.600020 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:39.010421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.010392 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl"] Nov 03 19:19:39.013619 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:19:39.013585 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod858e40ab_0455_4e17_bc4d_ff6163b3e750.slice/crio-e2de5494b44c8de8ff5ea8842b4ff58f578a96768e3c7a1aeaea276dc80f0db2 WatchSource:0}: Error finding container e2de5494b44c8de8ff5ea8842b4ff58f578a96768e3c7a1aeaea276dc80f0db2: Status 404 returned error can't find the container with id e2de5494b44c8de8ff5ea8842b4ff58f578a96768e3c7a1aeaea276dc80f0db2 Nov 03 19:19:39.134484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.134455 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:19:39.134765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.134746 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:19:39.135429 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.135411 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:19:39.274254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.274185 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7edbf57b-2dae-4e0f-8682-b08504feca4f" path="/var/lib/kubelet/pods/7edbf57b-2dae-4e0f-8682-b08504feca4f/volumes" Nov 03 19:19:39.856908 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.856866 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" event={"ID":"858e40ab-0455-4e17-bc4d-ff6163b3e750","Type":"ContainerStarted","Data":"5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed"} Nov 03 19:19:39.856908 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.856911 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" event={"ID":"858e40ab-0455-4e17-bc4d-ff6163b3e750","Type":"ContainerStarted","Data":"e2de5494b44c8de8ff5ea8842b4ff58f578a96768e3c7a1aeaea276dc80f0db2"} Nov 03 19:19:39.966890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:39.966836 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" podStartSLOduration=1.9668204280000001 podStartE2EDuration="1.966820428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:19:39.966465967 +0000 UTC m=+2454.847882068" watchObservedRunningTime="2026-11-03 19:19:39.966820428 +0000 UTC m=+2454.848236530" Nov 03 19:19:40.600338 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:40.600308 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:41.674540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:41.674502 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:43.105753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:43.105719 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:43.349453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:43.349410 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:43.555067 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:43.554980 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:44.036756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:44.036724 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:44.766959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:44.766922 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:45.134566 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.134537 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:45.347278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.347235 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:45.849056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.849021 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r"] Nov 03 19:19:45.853334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.853313 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.877856 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.877830 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.877976 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.877866 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.878057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.878039 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4r8w\" (UniqueName: \"kubernetes.io/projected/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-kube-api-access-l4r8w\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.907720 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.907654 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r"] Nov 03 19:19:45.951127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.951104 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:19:45.951268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.951155 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:19:45.979088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.979056 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4r8w\" (UniqueName: \"kubernetes.io/projected/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-kube-api-access-l4r8w\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.979246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.979187 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.979246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.979220 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.979827 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.979805 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:45.981591 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:45.981569 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:46.036748 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:46.036720 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4r8w\" (UniqueName: \"kubernetes.io/projected/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-kube-api-access-l4r8w\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:46.171282 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:46.171215 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:46.206513 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:46.206482 5988 scope.go:122] "RemoveContainer" containerID="1550da91a84b9dfabdcdfd4a007f068ea25ce4e962a9a368bc7179aa3020a197" Nov 03 19:19:46.217079 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:46.217049 5988 scope.go:122] "RemoveContainer" containerID="37ff6cf674e003333da3a661ec0bf3789b972b7ee3f010477e7eaaa9f8eb6a6c" Nov 03 19:19:46.544605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:46.544578 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r"] Nov 03 19:19:46.547103 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:19:46.547067 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf92f31_a3bf_40cb_b47e_edacdd8e5d78.slice/crio-e813900d43ad113a6d489e41e210e277d2388b8eabab76b0366ef4b3fea1b6ca WatchSource:0}: Error finding container e813900d43ad113a6d489e41e210e277d2388b8eabab76b0366ef4b3fea1b6ca: Status 404 returned error can't find the container with id e813900d43ad113a6d489e41e210e277d2388b8eabab76b0366ef4b3fea1b6ca Nov 03 19:19:46.915892 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:46.915857 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" event={"ID":"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78","Type":"ContainerStarted","Data":"900aec686587c82127611b55d2b3c5f2813b2419748bb0e3eeb0327b59751fdd"} Nov 03 19:19:46.916246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:46.915956 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" event={"ID":"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78","Type":"ContainerStarted","Data":"e813900d43ad113a6d489e41e210e277d2388b8eabab76b0366ef4b3fea1b6ca"} Nov 03 19:19:47.025507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:47.025460 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" podStartSLOduration=2.025447037 podStartE2EDuration="2.025447037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:19:47.024465937 +0000 UTC m=+2461.905882040" watchObservedRunningTime="2026-11-03 19:19:47.025447037 +0000 UTC m=+2461.906863138" Nov 03 19:19:47.924208 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:47.924172 5988 generic.go:356] "Generic (PLEG): container finished" podID="9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" containerID="900aec686587c82127611b55d2b3c5f2813b2419748bb0e3eeb0327b59751fdd" exitCode=0 Nov 03 19:19:47.924624 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:47.924253 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" event={"ID":"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78","Type":"ContainerDied","Data":"900aec686587c82127611b55d2b3c5f2813b2419748bb0e3eeb0327b59751fdd"} Nov 03 19:19:48.273042 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:48.272955 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:48.600224 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:48.600192 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:48.612823 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:48.612801 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:48.941996 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:48.941969 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:19:49.097929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.097902 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:49.216457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.216372 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-openshift-ca\") pod \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " Nov 03 19:19:49.216639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.216520 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-xks-ca\") pod \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " Nov 03 19:19:49.216639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.216553 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-kube-api-access-l4r8w\" (UniqueName: \"kubernetes.io/projected/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-kube-api-access-l4r8w\") pod \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\" (UID: \"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78\") " Nov 03 19:19:49.216819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.216790 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-openshift-ca" pod "9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" (UID: "9bf92f31-a3bf-40cb-b47e-edacdd8e5d78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:19:49.217257 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.217232 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:49.219012 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.218985 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-xks-ca" pod "9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" (UID: "9bf92f31-a3bf-40cb-b47e-edacdd8e5d78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:19:49.222160 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.222130 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-kube-api-access-l4r8w" pod "9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" (UID: "9bf92f31-a3bf-40cb-b47e-edacdd8e5d78"). InnerVolumeSpecName "kube-api-access-l4r8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:19:49.319140 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.319101 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:49.319292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.319146 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4r8w\" (UniqueName: \"kubernetes.io/projected/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78-kube-api-access-l4r8w\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:19:49.938881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.938852 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" Nov 03 19:19:49.939030 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.938872 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r" event={"ID":"9bf92f31-a3bf-40cb-b47e-edacdd8e5d78","Type":"ContainerDied","Data":"e813900d43ad113a6d489e41e210e277d2388b8eabab76b0366ef4b3fea1b6ca"} Nov 03 19:19:49.939030 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:49.938938 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e813900d43ad113a6d489e41e210e277d2388b8eabab76b0366ef4b3fea1b6ca" Nov 03 19:19:50.062630 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:50.062595 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:50.276124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:50.276044 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:50.484816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:50.484783 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:52.921081 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:52.921049 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:55.248462 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:55.248425 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:55.462032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:55.461994 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:55.937279 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:19:55.937246 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:19:56.282158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:56.282067 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:57.615930 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:57.615891 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:57.821993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:57.821961 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:19:58.022825 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:19:58.022738 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:00.730713 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:00.730658 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:20:02.504475 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:02.504439 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:06.477303 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:06.477256 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:07.930518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:07.930478 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:08.151459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:08.151426 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:08.346486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:08.346454 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv"] Nov 03 19:20:08.364225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:08.364196 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:08.392581 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:08.392554 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a4cacd-hqsdv"] Nov 03 19:20:08.849164 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:08.849113 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:09.272650 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:09.272578 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed35ebff-edce-48ce-8eb1-fdc2304a1a94" path="/var/lib/kubelet/pods/ed35ebff-edce-48ce-8eb1-fdc2304a1a94/volumes" Nov 03 19:20:11.239550 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:11.239501 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:11.447572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:11.447531 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:13.006176 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:13.006147 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:14.820672 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:14.820588 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:15.032228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:15.032194 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:15.126948 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:15.126862 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8"] Nov 03 19:20:15.127154 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:15.127127 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" containerID="cri-o://e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274" gracePeriod=30 Nov 03 19:20:15.236255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:15.236223 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:17.619470 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:17.619421 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:20.389133 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.389103 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:20:20.490074 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.489987 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-socket\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490074 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490063 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-credential-socket\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490115 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istiod-ca-cert\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490245 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-token\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-token\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490316 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-data\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490337 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-socket" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:20.490457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490385 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-certs\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490388 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-credential-socket" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:20.490457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490442 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-podinfo\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490533 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-envoy\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490575 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-kube-api-access-fm4p4\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-kube-api-access-fm4p4\") pod \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\" (UID: \"f30f3b22-0bce-490e-ba6c-d318c9c8b989\") " Nov 03 19:20:20.490757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490577 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istiod-ca-cert" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:20.490757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490584 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-data" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:20.490757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.490661 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-certs" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:20.491177 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.491151 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.491177 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.491179 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.491398 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.491195 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.491398 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.491209 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.491398 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.491223 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.492554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.492528 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-token" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:20.492728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.492704 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-kube-api-access-fm4p4" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "kube-api-access-fm4p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:20.492803 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.492719 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-podinfo" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:20:20.492843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.492800 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-envoy" pod "f30f3b22-0bce-490e-ba6c-d318c9c8b989" (UID: "f30f3b22-0bce-490e-ba6c-d318c9c8b989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:20:20.592409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.592371 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.592409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.592405 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm4p4\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-kube-api-access-fm4p4\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.592409 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.592416 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:20.592644 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:20.592425 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30f3b22-0bce-490e-ba6c-d318c9c8b989-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:21.134013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.133980 5988 generic.go:356] "Generic (PLEG): container finished" podID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerID="e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274" exitCode=0 Nov 03 19:20:21.134190 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.134023 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" Nov 03 19:20:21.134190 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.134057 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" event={"ID":"f30f3b22-0bce-490e-ba6c-d318c9c8b989","Type":"ContainerDied","Data":"e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274"} Nov 03 19:20:21.134190 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.134110 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8" event={"ID":"f30f3b22-0bce-490e-ba6c-d318c9c8b989","Type":"ContainerDied","Data":"8bc00074117221d9dacd5f0dcbd4eb65e35da8d1d784700e51aba797a7fe2382"} Nov 03 19:20:21.134190 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.134137 5988 scope.go:122] "RemoveContainer" containerID="e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274" Nov 03 19:20:21.152780 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.152760 5988 scope.go:122] "RemoveContainer" containerID="e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274" Nov 03 19:20:21.153094 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:20:21.153069 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274\": container with ID starting with e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274 not found: ID does not exist" containerID="e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274" Nov 03 19:20:21.153195 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.153097 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274"} err="failed to get container status \"e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274\": rpc error: code = NotFound desc = could not find container \"e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274\": container with ID starting with e11e517d7fa2322c539776fc8ed36a5f19411bef152f6e8df70d05ad305e8274 not found: ID does not exist" Nov 03 19:20:21.335630 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.335595 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8"] Nov 03 19:20:21.381494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:21.381464 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a4cacd-openshift-default-559dcbbf8c-4qtm8"] Nov 03 19:20:22.202300 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:22.202270 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:23.272741 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:23.272672 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" path="/var/lib/kubelet/pods/f30f3b22-0bce-490e-ba6c-d318c9c8b989/volumes" Nov 03 19:20:23.444977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:23.444944 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:25.936798 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:20:25.936764 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:20:26.759356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:26.759323 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:28.464955 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:28.464916 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:28.669437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:28.669408 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:30.779977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:30.779943 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:32.250330 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:32.250296 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:32.525763 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:32.525656 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:32.688790 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:32.688752 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r"] Nov 03 19:20:32.730357 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:32.730322 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:32.735064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:32.735035 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-58ac09-sf89r"] Nov 03 19:20:33.169302 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:33.169266 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:33.273627 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:33.273596 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" path="/var/lib/kubelet/pods/9bf92f31-a3bf-40cb-b47e-edacdd8e5d78/volumes" Nov 03 19:20:37.314404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:37.314372 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:39.291061 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:39.291021 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:20:39.455055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:39.455022 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:39.680468 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:39.680438 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:39.943728 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:39.943614 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:40.358581 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.358551 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:40.728281 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.728199 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd"] Nov 03 19:20:40.728849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.728829 5988 cpu_manager.go:414] "containerMap: removing container" podUID="9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" containerName="revoke-keys" Nov 03 19:20:40.728970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.728850 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" containerName="revoke-keys" Nov 03 19:20:40.728970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.728922 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" Nov 03 19:20:40.728970 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.728931 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" Nov 03 19:20:40.729129 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.729055 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf92f31-a3bf-40cb-b47e-edacdd8e5d78" containerName="revoke-keys" Nov 03 19:20:40.729129 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.729080 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30f3b22-0bce-490e-ba6c-d318c9c8b989" containerName="istio-proxy" Nov 03 19:20:40.732369 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.732349 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.826143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.826115 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c6058-4751-4130-9696-34065390feed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.826275 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.826260 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87q8c\" (UniqueName: \"kubernetes.io/projected/3b4c6058-4751-4130-9696-34065390feed-kube-api-access-87q8c\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.826323 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.826311 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4c6058-4751-4130-9696-34065390feed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.829929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.829903 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:20:40.830780 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.830766 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:20:40.832220 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.832203 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd"] Nov 03 19:20:40.927292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.927266 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87q8c\" (UniqueName: \"kubernetes.io/projected/3b4c6058-4751-4130-9696-34065390feed-kube-api-access-87q8c\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.927443 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.927313 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4c6058-4751-4130-9696-34065390feed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.927443 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.927397 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c6058-4751-4130-9696-34065390feed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.928165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.928146 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c6058-4751-4130-9696-34065390feed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.929756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.929736 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4c6058-4751-4130-9696-34065390feed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:40.984995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:40.984930 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87q8c\" (UniqueName: \"kubernetes.io/projected/3b4c6058-4751-4130-9696-34065390feed-kube-api-access-87q8c\") pod \"maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:41.046074 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:41.046038 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:41.415217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:41.415195 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd"] Nov 03 19:20:41.417508 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:20:41.417479 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4c6058_4751_4130_9696_34065390feed.slice/crio-8f61b9756ffbe7e3353ed44e815e6ab4504f6bdf8f2aedf388e8491d513d7e1a WatchSource:0}: Error finding container 8f61b9756ffbe7e3353ed44e815e6ab4504f6bdf8f2aedf388e8491d513d7e1a: Status 404 returned error can't find the container with id 8f61b9756ffbe7e3353ed44e815e6ab4504f6bdf8f2aedf388e8491d513d7e1a Nov 03 19:20:42.164819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:42.164785 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:42.263793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:42.263756 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" event={"ID":"3b4c6058-4751-4130-9696-34065390feed","Type":"ContainerStarted","Data":"9c0841f6213c48e951522733763ea5f17c0a8ddde4cc054a09eb362ef79bb39c"} Nov 03 19:20:42.263967 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:42.263818 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" event={"ID":"3b4c6058-4751-4130-9696-34065390feed","Type":"ContainerStarted","Data":"8f61b9756ffbe7e3353ed44e815e6ab4504f6bdf8f2aedf388e8491d513d7e1a"} Nov 03 19:20:42.373091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:42.373046 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" podStartSLOduration=2.373033319 podStartE2EDuration="2.373033319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:42.370179562 +0000 UTC m=+2517.251595667" watchObservedRunningTime="2026-11-03 19:20:42.373033319 +0000 UTC m=+2517.254449414" Nov 03 19:20:43.273885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:43.273849 5988 generic.go:356] "Generic (PLEG): container finished" podID="3b4c6058-4751-4130-9696-34065390feed" containerID="9c0841f6213c48e951522733763ea5f17c0a8ddde4cc054a09eb362ef79bb39c" exitCode=0 Nov 03 19:20:43.275183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:43.275157 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" event={"ID":"3b4c6058-4751-4130-9696-34065390feed","Type":"ContainerDied","Data":"9c0841f6213c48e951522733763ea5f17c0a8ddde4cc054a09eb362ef79bb39c"} Nov 03 19:20:43.894259 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:43.894217 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:44.415854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.415826 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:44.475553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.475528 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4c6058-4751-4130-9696-34065390feed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c6058-4751-4130-9696-34065390feed-maas-api-openshift-ca\") pod \"3b4c6058-4751-4130-9696-34065390feed\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " Nov 03 19:20:44.475673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.475600 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4c6058-4751-4130-9696-34065390feed-kube-api-access-87q8c\" (UniqueName: \"kubernetes.io/projected/3b4c6058-4751-4130-9696-34065390feed-kube-api-access-87q8c\") pod \"3b4c6058-4751-4130-9696-34065390feed\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " Nov 03 19:20:44.475757 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.475718 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b4c6058-4751-4130-9696-34065390feed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4c6058-4751-4130-9696-34065390feed-maas-api-xks-ca\") pod \"3b4c6058-4751-4130-9696-34065390feed\" (UID: \"3b4c6058-4751-4130-9696-34065390feed\") " Nov 03 19:20:44.475954 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.475933 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4c6058-4751-4130-9696-34065390feed-maas-api-openshift-ca" pod "3b4c6058-4751-4130-9696-34065390feed" (UID: "3b4c6058-4751-4130-9696-34065390feed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:20:44.476140 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.476127 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4c6058-4751-4130-9696-34065390feed-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:44.477601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.477581 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b4c6058-4751-4130-9696-34065390feed-maas-api-xks-ca" pod "3b4c6058-4751-4130-9696-34065390feed" (UID: "3b4c6058-4751-4130-9696-34065390feed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:20:44.477705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.477640 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4c6058-4751-4130-9696-34065390feed-kube-api-access-87q8c" pod "3b4c6058-4751-4130-9696-34065390feed" (UID: "3b4c6058-4751-4130-9696-34065390feed"). InnerVolumeSpecName "kube-api-access-87q8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:20:44.577307 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.577279 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87q8c\" (UniqueName: \"kubernetes.io/projected/3b4c6058-4751-4130-9696-34065390feed-kube-api-access-87q8c\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:44.577443 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:44.577309 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4c6058-4751-4130-9696-34065390feed-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:20:45.218158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:45.218109 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h"] Nov 03 19:20:45.288863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:45.288835 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" Nov 03 19:20:45.288863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:45.288853 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd" event={"ID":"3b4c6058-4751-4130-9696-34065390feed","Type":"ContainerDied","Data":"8f61b9756ffbe7e3353ed44e815e6ab4504f6bdf8f2aedf388e8491d513d7e1a"} Nov 03 19:20:45.289095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:45.288894 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f61b9756ffbe7e3353ed44e815e6ab4504f6bdf8f2aedf388e8491d513d7e1a" Nov 03 19:20:45.406244 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:45.406209 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:46.283484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:46.283451 5988 scope.go:122] "RemoveContainer" containerID="0b1b76108268b15da0faa5920ead142f47c8c0781fc0e601d77cd6aa579cb741" Nov 03 19:20:46.293588 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:46.293571 5988 scope.go:122] "RemoveContainer" containerID="0f19cd8a7952f3482bc699c3334cef70a4b55715d9f2f9e741eb0e99213b7dd2" Nov 03 19:20:46.304241 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:46.304210 5988 scope.go:122] "RemoveContainer" containerID="812afa04d4172e421950c3c8f0e19cae4584db106b5634dd852c20f2890491ef" Nov 03 19:20:46.808372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:46.808339 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:51.338885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:51.338847 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:53.373281 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:53.373241 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:53.578371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:53.578330 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:55.593623 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:55.593582 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:55.915325 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:20:55.915231 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:20:55.937904 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:20:55.937869 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:21:00.239298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.239252 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" containerID="cri-o://7ddd0d3e47724a9077721df62ddbb26c709a2a6435fdf837d8f9cc71b798f5ff" gracePeriod=45 Nov 03 19:21:00.346253 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.346215 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": dial tcp 10.134.0.41:8000: connect: connection refused" Nov 03 19:21:00.394662 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.394626 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:00.446471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.446435 5988 generic.go:356] "Generic (PLEG): container finished" podID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerID="7ddd0d3e47724a9077721df62ddbb26c709a2a6435fdf837d8f9cc71b798f5ff" exitCode=0 Nov 03 19:21:00.446616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.446501 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" event={"ID":"6ed9f5e1-ead3-4a19-9ed1-25687b8db664","Type":"ContainerDied","Data":"7ddd0d3e47724a9077721df62ddbb26c709a2a6435fdf837d8f9cc71b798f5ff"} Nov 03 19:21:00.521311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.521287 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 19:21:00.665579 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.665549 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-model-cache\") pod \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " Nov 03 19:21:00.665797 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.665596 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-kube-api-access-4wgcb\" (UniqueName: \"kubernetes.io/projected/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-kube-api-access-4wgcb\") pod \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " Nov 03 19:21:00.665797 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.665629 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tls-certs\") pod \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " Nov 03 19:21:00.665797 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.665665 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-dshm\") pod \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " Nov 03 19:21:00.665797 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.665750 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-home\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-home\") pod \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " Nov 03 19:21:00.666040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.665882 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tmp-dir\") pod \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\" (UID: \"6ed9f5e1-ead3-4a19-9ed1-25687b8db664\") " Nov 03 19:21:00.666040 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.665908 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-model-cache" pod "6ed9f5e1-ead3-4a19-9ed1-25687b8db664" (UID: "6ed9f5e1-ead3-4a19-9ed1-25687b8db664"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.666161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.666040 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-home" pod "6ed9f5e1-ead3-4a19-9ed1-25687b8db664" (UID: "6ed9f5e1-ead3-4a19-9ed1-25687b8db664"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.666161 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.666112 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tmp-dir" pod "6ed9f5e1-ead3-4a19-9ed1-25687b8db664" (UID: "6ed9f5e1-ead3-4a19-9ed1-25687b8db664"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.666341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.666321 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.666414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.666344 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.666414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.666359 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.667925 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.667901 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-dshm" pod "6ed9f5e1-ead3-4a19-9ed1-25687b8db664" (UID: "6ed9f5e1-ead3-4a19-9ed1-25687b8db664"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:21:00.668028 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.667949 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-kube-api-access-4wgcb" pod "6ed9f5e1-ead3-4a19-9ed1-25687b8db664" (UID: "6ed9f5e1-ead3-4a19-9ed1-25687b8db664"). InnerVolumeSpecName "kube-api-access-4wgcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:00.668028 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.667973 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tls-certs" pod "6ed9f5e1-ead3-4a19-9ed1-25687b8db664" (UID: "6ed9f5e1-ead3-4a19-9ed1-25687b8db664"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:00.767295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.767217 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wgcb\" (UniqueName: \"kubernetes.io/projected/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-kube-api-access-4wgcb\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.767295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.767251 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:00.767295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:00.767266 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ed9f5e1-ead3-4a19-9ed1-25687b8db664-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:01.320817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.320783 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:01.455348 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.455315 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" Nov 03 19:21:01.455348 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.455339 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h" event={"ID":"6ed9f5e1-ead3-4a19-9ed1-25687b8db664","Type":"ContainerDied","Data":"772ab499c4cafd97762edaa5938c56db75e0068e274faf9e51d07c13db9eed2b"} Nov 03 19:21:01.455594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.455389 5988 scope.go:122] "RemoveContainer" containerID="7ddd0d3e47724a9077721df62ddbb26c709a2a6435fdf837d8f9cc71b798f5ff" Nov 03 19:21:01.525587 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.525548 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:01.609403 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.609366 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h"] Nov 03 19:21:01.655909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.655870 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2e052a/simulator-w4-2e052a-kserve-7c4c9857f4-rp27h"] Nov 03 19:21:01.801765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:01.801730 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:21:03.272799 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:03.272767 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" path="/var/lib/kubelet/pods/6ed9f5e1-ead3-4a19-9ed1-25687b8db664/volumes" Nov 03 19:21:05.005957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:05.005919 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:08.551418 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:08.551386 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:08.751754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:08.751721 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:09.607175 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:09.607134 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:14.098294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:14.098262 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:16.637348 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.637306 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs"] Nov 03 19:21:16.637960 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.637939 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3b4c6058-4751-4130-9696-34065390feed" containerName="revoke-keys" Nov 03 19:21:16.638069 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.637962 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4c6058-4751-4130-9696-34065390feed" containerName="revoke-keys" Nov 03 19:21:16.638069 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.638032 5988 cpu_manager.go:414] "containerMap: removing container" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" Nov 03 19:21:16.638069 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.638052 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" Nov 03 19:21:16.638225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.638178 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed9f5e1-ead3-4a19-9ed1-25687b8db664" containerName="main" Nov 03 19:21:16.638225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.638191 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4c6058-4751-4130-9696-34065390feed" containerName="revoke-keys" Nov 03 19:21:16.641862 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.641843 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.692908 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.692868 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:21:16.693108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.693086 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:21:16.741712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.741669 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs"] Nov 03 19:21:16.749997 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.749966 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.750127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.750023 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgfx\" (UniqueName: \"kubernetes.io/projected/4284fd88-cfbc-403c-ae7d-e393d0e82df2-kube-api-access-msgfx\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.750169 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.750145 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.851602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.851568 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.851782 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.851714 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgfx\" (UniqueName: \"kubernetes.io/projected/4284fd88-cfbc-403c-ae7d-e393d0e82df2-kube-api-access-msgfx\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.851862 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.851812 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.852479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.852454 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.853965 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.853944 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.909356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.909295 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgfx\" (UniqueName: \"kubernetes.io/projected/4284fd88-cfbc-403c-ae7d-e393d0e82df2-kube-api-access-msgfx\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:16.955443 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:16.955418 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:17.321572 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:17.321547 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs"] Nov 03 19:21:17.326789 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:21:17.326757 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4284fd88_cfbc_403c_ae7d_e393d0e82df2.slice/crio-b280e3adb153351de392eb8dfc41967e4ea73d1214a9f56428ac9eadc6c9c609 WatchSource:0}: Error finding container b280e3adb153351de392eb8dfc41967e4ea73d1214a9f56428ac9eadc6c9c609: Status 404 returned error can't find the container with id b280e3adb153351de392eb8dfc41967e4ea73d1214a9f56428ac9eadc6c9c609 Nov 03 19:21:17.555545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:17.555509 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" event={"ID":"4284fd88-cfbc-403c-ae7d-e393d0e82df2","Type":"ContainerStarted","Data":"b280e3adb153351de392eb8dfc41967e4ea73d1214a9f56428ac9eadc6c9c609"} Nov 03 19:21:18.563316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:18.563221 5988 generic.go:356] "Generic (PLEG): container finished" podID="4284fd88-cfbc-403c-ae7d-e393d0e82df2" containerID="338c614bd6cf54300751de1dab2a927561bfbf6613891095f32750fe58b175a2" exitCode=0 Nov 03 19:21:18.563717 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:18.563304 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" event={"ID":"4284fd88-cfbc-403c-ae7d-e393d0e82df2","Type":"ContainerDied","Data":"338c614bd6cf54300751de1dab2a927561bfbf6613891095f32750fe58b175a2"} Nov 03 19:21:18.941176 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:18.941137 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:19.720617 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.720589 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:19.785464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.785430 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4284fd88-cfbc-403c-ae7d-e393d0e82df2-kube-api-access-msgfx\" (UniqueName: \"kubernetes.io/projected/4284fd88-cfbc-403c-ae7d-e393d0e82df2-kube-api-access-msgfx\") pod \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " Nov 03 19:21:19.785670 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.785592 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-openshift-ca\") pod \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " Nov 03 19:21:19.785670 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.785654 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-xks-ca\") pod \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\" (UID: \"4284fd88-cfbc-403c-ae7d-e393d0e82df2\") " Nov 03 19:21:19.786000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.785970 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-openshift-ca" pod "4284fd88-cfbc-403c-ae7d-e393d0e82df2" (UID: "4284fd88-cfbc-403c-ae7d-e393d0e82df2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:19.786247 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.786206 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:19.787706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.787668 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-xks-ca" pod "4284fd88-cfbc-403c-ae7d-e393d0e82df2" (UID: "4284fd88-cfbc-403c-ae7d-e393d0e82df2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:19.787813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.787764 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4284fd88-cfbc-403c-ae7d-e393d0e82df2-kube-api-access-msgfx" pod "4284fd88-cfbc-403c-ae7d-e393d0e82df2" (UID: "4284fd88-cfbc-403c-ae7d-e393d0e82df2"). InnerVolumeSpecName "kube-api-access-msgfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:19.887501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.887414 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4284fd88-cfbc-403c-ae7d-e393d0e82df2-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:19.887501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:19.887447 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msgfx\" (UniqueName: \"kubernetes.io/projected/4284fd88-cfbc-403c-ae7d-e393d0e82df2-kube-api-access-msgfx\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:20.583738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:20.583707 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" Nov 03 19:21:20.583738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:20.583711 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs" event={"ID":"4284fd88-cfbc-403c-ae7d-e393d0e82df2","Type":"ContainerDied","Data":"b280e3adb153351de392eb8dfc41967e4ea73d1214a9f56428ac9eadc6c9c609"} Nov 03 19:21:20.583955 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:20.583778 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b280e3adb153351de392eb8dfc41967e4ea73d1214a9f56428ac9eadc6c9c609" Nov 03 19:21:22.720167 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:22.720123 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:22.921547 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:22.921509 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:23.345331 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:23.345297 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:23.727814 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:23.727703 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:25.937381 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:21:25.937317 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:21:27.790732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.790673 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm"] Nov 03 19:21:27.791300 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.791280 5988 cpu_manager.go:414] "containerMap: removing container" podUID="4284fd88-cfbc-403c-ae7d-e393d0e82df2" containerName="revoke-keys" Nov 03 19:21:27.791358 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.791302 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4284fd88-cfbc-403c-ae7d-e393d0e82df2" containerName="revoke-keys" Nov 03 19:21:27.791426 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.791416 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4284fd88-cfbc-403c-ae7d-e393d0e82df2" containerName="revoke-keys" Nov 03 19:21:27.794816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.794793 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.846987 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.846966 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:21:27.849363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.849342 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:21:27.889420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.889397 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q82np\" (UniqueName: \"kubernetes.io/projected/e4fe2b11-c762-4290-a841-e450d67b71cc-kube-api-access-q82np\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.889541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.889444 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.889696 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.889577 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.897726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.897701 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm"] Nov 03 19:21:27.990586 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.990553 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.990748 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.990612 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.990802 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.990756 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q82np\" (UniqueName: \"kubernetes.io/projected/e4fe2b11-c762-4290-a841-e450d67b71cc-kube-api-access-q82np\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.991356 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.991337 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:27.992982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:27.992964 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:28.048195 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.048134 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q82np\" (UniqueName: \"kubernetes.io/projected/e4fe2b11-c762-4290-a841-e450d67b71cc-kube-api-access-q82np\") pod \"maas-api-revoke-sub-7688d859-g2rkm\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:28.108044 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.108009 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:28.272958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.272928 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9"] Nov 03 19:21:28.494452 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.494416 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm"] Nov 03 19:21:28.497251 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.497216 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:21:28.499106 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:21:28.499073 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fe2b11_c762_4290_a841_e450d67b71cc.slice/crio-4ba0d9aff3ff9f4c9b3ab7158fb20cd66ce7fdf26192ecdc7ef42f696b0b6466 WatchSource:0}: Error finding container 4ba0d9aff3ff9f4c9b3ab7158fb20cd66ce7fdf26192ecdc7ef42f696b0b6466: Status 404 returned error can't find the container with id 4ba0d9aff3ff9f4c9b3ab7158fb20cd66ce7fdf26192ecdc7ef42f696b0b6466 Nov 03 19:21:28.598734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.598676 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:28.649225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.649186 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" event={"ID":"e4fe2b11-c762-4290-a841-e450d67b71cc","Type":"ContainerStarted","Data":"4ba0d9aff3ff9f4c9b3ab7158fb20cd66ce7fdf26192ecdc7ef42f696b0b6466"} Nov 03 19:21:28.803010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:28.802977 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:29.672708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:29.672014 5988 generic.go:356] "Generic (PLEG): container finished" podID="e4fe2b11-c762-4290-a841-e450d67b71cc" containerID="cfd47c64fdd6de04fca92fb2d5b01db597f6e8fdc2c0eb2acb28dc8ff39a4237" exitCode=0 Nov 03 19:21:29.672708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:29.672118 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" event={"ID":"e4fe2b11-c762-4290-a841-e450d67b71cc","Type":"ContainerDied","Data":"cfd47c64fdd6de04fca92fb2d5b01db597f6e8fdc2c0eb2acb28dc8ff39a4237"} Nov 03 19:21:29.816404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:29.816367 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:30.018926 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.018836 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:30.813510 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.813488 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:30.823524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.823495 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fe2b11-c762-4290-a841-e450d67b71cc-kube-api-access-q82np\" (UniqueName: \"kubernetes.io/projected/e4fe2b11-c762-4290-a841-e450d67b71cc-kube-api-access-q82np\") pod \"e4fe2b11-c762-4290-a841-e450d67b71cc\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " Nov 03 19:21:30.823966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.823710 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-openshift-ca\") pod \"e4fe2b11-c762-4290-a841-e450d67b71cc\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " Nov 03 19:21:30.823966 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.823754 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-xks-ca\") pod \"e4fe2b11-c762-4290-a841-e450d67b71cc\" (UID: \"e4fe2b11-c762-4290-a841-e450d67b71cc\") " Nov 03 19:21:30.824096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.823996 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-openshift-ca" pod "e4fe2b11-c762-4290-a841-e450d67b71cc" (UID: "e4fe2b11-c762-4290-a841-e450d67b71cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:30.824249 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.824230 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.825856 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.825824 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fe2b11-c762-4290-a841-e450d67b71cc-kube-api-access-q82np" pod "e4fe2b11-c762-4290-a841-e450d67b71cc" (UID: "e4fe2b11-c762-4290-a841-e450d67b71cc"). InnerVolumeSpecName "kube-api-access-q82np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:30.826958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.826938 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-xks-ca" pod "e4fe2b11-c762-4290-a841-e450d67b71cc" (UID: "e4fe2b11-c762-4290-a841-e450d67b71cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:21:30.926032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.925961 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fe2b11-c762-4290-a841-e450d67b71cc-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:30.926032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:30.925994 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q82np\" (UniqueName: \"kubernetes.io/projected/e4fe2b11-c762-4290-a841-e450d67b71cc-kube-api-access-q82np\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:31.685655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:31.685626 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" Nov 03 19:21:31.685655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:31.685644 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm" event={"ID":"e4fe2b11-c762-4290-a841-e450d67b71cc","Type":"ContainerDied","Data":"4ba0d9aff3ff9f4c9b3ab7158fb20cd66ce7fdf26192ecdc7ef42f696b0b6466"} Nov 03 19:21:31.685862 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:31.685704 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba0d9aff3ff9f4c9b3ab7158fb20cd66ce7fdf26192ecdc7ef42f696b0b6466" Nov 03 19:21:33.211036 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:33.210997 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:33.273364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:33.273310 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" podUID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" containerName="payload-pre-processing" containerID="cri-o://de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419" gracePeriod=25 Nov 03 19:21:37.732568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:37.732532 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:41.093977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:41.093938 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" podUID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:21:42.309474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:42.309435 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:46.345527 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:46.345497 5988 scope.go:122] "RemoveContainer" containerID="acf8a5460069dc349bb541f813d03a485d7df96c028f411782574bbe81a9b325" Nov 03 19:21:47.026268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:47.026233 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:51.094629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:51.094583 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" podUID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:21:51.609609 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:51.609573 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:53.534273 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.534202 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:21:53.673637 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.673596 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b89d93-3ebe-499e-9ed9-00770e4805c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b89d93-3ebe-499e-9ed9-00770e4805c3-plugins-config-volume\") pod \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " Nov 03 19:21:53.673871 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.673654 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b89d93-3ebe-499e-9ed9-00770e4805c3-kube-api-access-smnrh\" (UniqueName: \"kubernetes.io/projected/c8b89d93-3ebe-499e-9ed9-00770e4805c3-kube-api-access-smnrh\") pod \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\" (UID: \"c8b89d93-3ebe-499e-9ed9-00770e4805c3\") " Nov 03 19:21:53.674026 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.674001 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b89d93-3ebe-499e-9ed9-00770e4805c3-plugins-config-volume" pod "c8b89d93-3ebe-499e-9ed9-00770e4805c3" (UID: "c8b89d93-3ebe-499e-9ed9-00770e4805c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:21:53.675934 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.675909 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b89d93-3ebe-499e-9ed9-00770e4805c3-kube-api-access-smnrh" pod "c8b89d93-3ebe-499e-9ed9-00770e4805c3" (UID: "c8b89d93-3ebe-499e-9ed9-00770e4805c3"). InnerVolumeSpecName "kube-api-access-smnrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:21:53.775573 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.775535 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b89d93-3ebe-499e-9ed9-00770e4805c3-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:53.775573 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.775566 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smnrh\" (UniqueName: \"kubernetes.io/projected/c8b89d93-3ebe-499e-9ed9-00770e4805c3-kube-api-access-smnrh\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:21:53.820032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.819997 5988 generic.go:356] "Generic (PLEG): container finished" podID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" containerID="de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419" exitCode=0 Nov 03 19:21:53.820232 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.820197 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" event={"ID":"c8b89d93-3ebe-499e-9ed9-00770e4805c3","Type":"ContainerDied","Data":"de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419"} Nov 03 19:21:53.820286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.820252 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" event={"ID":"c8b89d93-3ebe-499e-9ed9-00770e4805c3","Type":"ContainerDied","Data":"d48d5f15beaeacf70cf27f8ef387aaa9af917643e927f2fc1c4cabd6a2ebbbcb"} Nov 03 19:21:53.820286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.820285 5988 scope.go:122] "RemoveContainer" containerID="de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419" Nov 03 19:21:53.820420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.820408 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9" Nov 03 19:21:53.830767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.830748 5988 scope.go:122] "RemoveContainer" containerID="de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419" Nov 03 19:21:53.831070 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:21:53.831048 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419\": container with ID starting with de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419 not found: ID does not exist" containerID="de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419" Nov 03 19:21:53.831127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.831080 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419"} err="failed to get container status \"de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419\": rpc error: code = NotFound desc = could not find container \"de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419\": container with ID starting with de31faefc40251ad44d8d3ae8f43616f14cad05df251b5cadd3c4c2818b26419 not found: ID does not exist" Nov 03 19:21:53.975417 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.975376 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9"] Nov 03 19:21:53.981899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:53.981864 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3d74f801-67d6d795dc-65gv9"] Nov 03 19:21:55.272491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:55.272453 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" path="/var/lib/kubelet/pods/c8b89d93-3ebe-499e-9ed9-00770e4805c3/volumes" Nov 03 19:21:55.936273 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:21:55.936235 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:21:55.943176 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:55.943151 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:56.151996 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:56.151959 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:21:58.712265 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:21:58.712230 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:22:00.057177 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.057142 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7"] Nov 03 19:22:00.057676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.057661 5988 cpu_manager.go:414] "containerMap: removing container" podUID="e4fe2b11-c762-4290-a841-e450d67b71cc" containerName="revoke-keys" Nov 03 19:22:00.057676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.057676 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fe2b11-c762-4290-a841-e450d67b71cc" containerName="revoke-keys" Nov 03 19:22:00.057783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.057729 5988 cpu_manager.go:414] "containerMap: removing container" podUID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" containerName="payload-pre-processing" Nov 03 19:22:00.057783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.057734 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" containerName="payload-pre-processing" Nov 03 19:22:00.057849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.057831 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b89d93-3ebe-499e-9ed9-00770e4805c3" containerName="payload-pre-processing" Nov 03 19:22:00.057849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.057844 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fe2b11-c762-4290-a841-e450d67b71cc" containerName="revoke-keys" Nov 03 19:22:00.062277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.062260 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.159864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.159831 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:22:00.160035 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.159946 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:22:00.162393 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.162373 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7"] Nov 03 19:22:00.249959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.249926 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lkqd\" (UniqueName: \"kubernetes.io/projected/f4c8cf53-a227-422a-91c1-61193a15b2e9-kube-api-access-9lkqd\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.250131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.249974 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.250185 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.250157 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.351643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.351603 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.351880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.351722 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lkqd\" (UniqueName: \"kubernetes.io/projected/f4c8cf53-a227-422a-91c1-61193a15b2e9-kube-api-access-9lkqd\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.351880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.351756 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.352376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.352357 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.353989 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.353970 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.409798 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.409765 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lkqd\" (UniqueName: \"kubernetes.io/projected/f4c8cf53-a227-422a-91c1-61193a15b2e9-kube-api-access-9lkqd\") pod \"maas-api-revoke-sub-bd72aefc-td5j7\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.675433 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.675348 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:00.805790 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:00.805751 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:01.044038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:01.044008 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7"] Nov 03 19:22:01.046077 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:22:01.046047 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c8cf53_a227_422a_91c1_61193a15b2e9.slice/crio-08437e6b338068da1a5e9a8d41051523c415c21b5cfc2cf9e85b11abd881554f WatchSource:0}: Error finding container 08437e6b338068da1a5e9a8d41051523c415c21b5cfc2cf9e85b11abd881554f: Status 404 returned error can't find the container with id 08437e6b338068da1a5e9a8d41051523c415c21b5cfc2cf9e85b11abd881554f Nov 03 19:22:01.876073 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:01.876043 5988 generic.go:356] "Generic (PLEG): container finished" podID="f4c8cf53-a227-422a-91c1-61193a15b2e9" containerID="f1c1cff0b360069fabe25999a6a0e7440926b97defafd79d2d99eec85bdfc4ac" exitCode=0 Nov 03 19:22:01.876395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:01.876130 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" event={"ID":"f4c8cf53-a227-422a-91c1-61193a15b2e9","Type":"ContainerDied","Data":"f1c1cff0b360069fabe25999a6a0e7440926b97defafd79d2d99eec85bdfc4ac"} Nov 03 19:22:01.876395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:01.876188 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" event={"ID":"f4c8cf53-a227-422a-91c1-61193a15b2e9","Type":"ContainerStarted","Data":"08437e6b338068da1a5e9a8d41051523c415c21b5cfc2cf9e85b11abd881554f"} Nov 03 19:22:03.025923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.025901 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:03.076670 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.076637 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c8cf53-a227-422a-91c1-61193a15b2e9-kube-api-access-9lkqd\" (UniqueName: \"kubernetes.io/projected/f4c8cf53-a227-422a-91c1-61193a15b2e9-kube-api-access-9lkqd\") pod \"f4c8cf53-a227-422a-91c1-61193a15b2e9\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " Nov 03 19:22:03.076863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.076780 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-openshift-ca\") pod \"f4c8cf53-a227-422a-91c1-61193a15b2e9\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " Nov 03 19:22:03.076936 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.076888 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-xks-ca\") pod \"f4c8cf53-a227-422a-91c1-61193a15b2e9\" (UID: \"f4c8cf53-a227-422a-91c1-61193a15b2e9\") " Nov 03 19:22:03.077295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.077266 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-openshift-ca" pod "f4c8cf53-a227-422a-91c1-61193a15b2e9" (UID: "f4c8cf53-a227-422a-91c1-61193a15b2e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:03.077607 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.077583 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.078982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.078958 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c8cf53-a227-422a-91c1-61193a15b2e9-kube-api-access-9lkqd" pod "f4c8cf53-a227-422a-91c1-61193a15b2e9" (UID: "f4c8cf53-a227-422a-91c1-61193a15b2e9"). InnerVolumeSpecName "kube-api-access-9lkqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:03.079278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.079257 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-xks-ca" pod "f4c8cf53-a227-422a-91c1-61193a15b2e9" (UID: "f4c8cf53-a227-422a-91c1-61193a15b2e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:03.179452 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.179362 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lkqd\" (UniqueName: \"kubernetes.io/projected/f4c8cf53-a227-422a-91c1-61193a15b2e9-kube-api-access-9lkqd\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.179452 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.179389 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4c8cf53-a227-422a-91c1-61193a15b2e9-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:03.889848 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.889816 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" Nov 03 19:22:03.890038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.889858 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7" event={"ID":"f4c8cf53-a227-422a-91c1-61193a15b2e9","Type":"ContainerDied","Data":"08437e6b338068da1a5e9a8d41051523c415c21b5cfc2cf9e85b11abd881554f"} Nov 03 19:22:03.890038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:03.889915 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08437e6b338068da1a5e9a8d41051523c415c21b5cfc2cf9e85b11abd881554f" Nov 03 19:22:04.637988 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.637956 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z"] Nov 03 19:22:04.638458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.638160 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" podUID="da384f09-39e5-4c50-872c-6cc981339aa0" containerName="maas-api" containerID="cri-o://7b5e0230458f463affa347f4d8af293134e73168fe9f8fd99602b18dc1b2276f" gracePeriod=30 Nov 03 19:22:04.800923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.800858 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:04.899229 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.899195 5988 generic.go:356] "Generic (PLEG): container finished" podID="da384f09-39e5-4c50-872c-6cc981339aa0" containerID="7b5e0230458f463affa347f4d8af293134e73168fe9f8fd99602b18dc1b2276f" exitCode=0 Nov 03 19:22:04.899363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.899265 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" event={"ID":"da384f09-39e5-4c50-872c-6cc981339aa0","Type":"ContainerDied","Data":"7b5e0230458f463affa347f4d8af293134e73168fe9f8fd99602b18dc1b2276f"} Nov 03 19:22:04.899363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.899314 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" event={"ID":"da384f09-39e5-4c50-872c-6cc981339aa0","Type":"ContainerDied","Data":"7cfe5e389006a8a11a760c2a5365b491b6a774f0971dd79624a1279b1be11f68"} Nov 03 19:22:04.899363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.899331 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cfe5e389006a8a11a760c2a5365b491b6a774f0971dd79624a1279b1be11f68" Nov 03 19:22:04.913786 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:04.913766 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 19:22:05.013204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.013161 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da384f09-39e5-4c50-872c-6cc981339aa0-kube-api-access-q4v5f\" (UniqueName: \"kubernetes.io/projected/da384f09-39e5-4c50-872c-6cc981339aa0-kube-api-access-q4v5f\") pod \"da384f09-39e5-4c50-872c-6cc981339aa0\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " Nov 03 19:22:05.013385 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.013293 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-metrics-certs\") pod \"da384f09-39e5-4c50-872c-6cc981339aa0\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " Nov 03 19:22:05.013490 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.013460 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da384f09-39e5-4c50-872c-6cc981339aa0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da384f09-39e5-4c50-872c-6cc981339aa0-openshift-service-ca\") pod \"da384f09-39e5-4c50-872c-6cc981339aa0\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " Nov 03 19:22:05.013554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.013504 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls\") pod \"da384f09-39e5-4c50-872c-6cc981339aa0\" (UID: \"da384f09-39e5-4c50-872c-6cc981339aa0\") " Nov 03 19:22:05.015143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.015110 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da384f09-39e5-4c50-872c-6cc981339aa0-openshift-service-ca" pod "da384f09-39e5-4c50-872c-6cc981339aa0" (UID: "da384f09-39e5-4c50-872c-6cc981339aa0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:05.017889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.017135 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da384f09-39e5-4c50-872c-6cc981339aa0-kube-api-access-q4v5f" pod "da384f09-39e5-4c50-872c-6cc981339aa0" (UID: "da384f09-39e5-4c50-872c-6cc981339aa0"). InnerVolumeSpecName "kube-api-access-q4v5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:05.023138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.023101 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-metrics-certs" pod "da384f09-39e5-4c50-872c-6cc981339aa0" (UID: "da384f09-39e5-4c50-872c-6cc981339aa0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:05.023138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.023116 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls" pod "da384f09-39e5-4c50-872c-6cc981339aa0" (UID: "da384f09-39e5-4c50-872c-6cc981339aa0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:22:05.115197 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.115167 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4v5f\" (UniqueName: \"kubernetes.io/projected/da384f09-39e5-4c50-872c-6cc981339aa0-kube-api-access-q4v5f\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:05.115197 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.115193 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:05.115373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.115203 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da384f09-39e5-4c50-872c-6cc981339aa0-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:05.115373 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.115212 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da384f09-39e5-4c50-872c-6cc981339aa0-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:05.905427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:05.905390 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z" Nov 03 19:22:06.061718 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:06.061677 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z"] Nov 03 19:22:06.108024 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:06.107993 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-2e052a-75f766dbf8-nwd7z"] Nov 03 19:22:06.291839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:06.291765 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:06.544710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:06.544580 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:06.748325 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:06.748288 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:06.841526 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:06.841452 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd"] Nov 03 19:22:06.844725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:06.844544 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2e052a-mf4cd"] Nov 03 19:22:07.272953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:07.272875 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4c6058-4751-4130-9696-34065390feed" path="/var/lib/kubelet/pods/3b4c6058-4751-4130-9696-34065390feed/volumes" Nov 03 19:22:07.273341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:07.273325 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da384f09-39e5-4c50-872c-6cc981339aa0" path="/var/lib/kubelet/pods/da384f09-39e5-4c50-872c-6cc981339aa0/volumes" Nov 03 19:22:07.403433 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:07.403400 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:11.386193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:11.386157 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:13.982332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:13.982291 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:14.199443 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:14.199408 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:14.236363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:14.236260 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt"] Nov 03 19:22:14.236638 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:14.236585 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" containerID="cri-o://8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20" gracePeriod=30 Nov 03 19:22:14.405974 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:14.405937 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:15.307245 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:15.307205 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:15.519640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:15.519599 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:16.004916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:16.004884 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:19.488760 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.488732 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 19:22:19.596934 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.596902 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-credential-socket\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.596987 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-certs\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597022 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-socket\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597097 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-kube-api-access-gct4c\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-kube-api-access-gct4c\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597136 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-podinfo\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597160 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-data\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597179 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-credential-socket" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:19.597295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597199 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istiod-ca-cert\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597236 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-token\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-token\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597295 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597273 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-envoy\") pod \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\" (UID: \"96fea62b-6fd4-40b2-8b7d-21b04f456fdd\") " Nov 03 19:22:19.597545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597331 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-certs" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:19.597789 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.597755 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-socket" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:19.598134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.598105 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-data" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:19.598293 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.598247 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istiod-ca-cert" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:22:19.600096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.599263 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.600096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.599291 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.600096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.599316 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.600096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.599331 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.600096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.599347 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.600405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.600350 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-envoy" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:22:19.600405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.600355 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-podinfo" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:22:19.600939 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.600909 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-kube-api-access-gct4c" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "kube-api-access-gct4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:19.609080 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.609026 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-token" pod "96fea62b-6fd4-40b2-8b7d-21b04f456fdd" (UID: "96fea62b-6fd4-40b2-8b7d-21b04f456fdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:22:19.700256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.700213 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gct4c\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-kube-api-access-gct4c\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.700256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.700249 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.700256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.700263 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.700514 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.700277 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fea62b-6fd4-40b2-8b7d-21b04f456fdd-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:22:19.998392 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.998302 5988 generic.go:356] "Generic (PLEG): container finished" podID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerID="8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20" exitCode=0 Nov 03 19:22:19.998392 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.998369 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" Nov 03 19:22:19.998587 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.998388 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" event={"ID":"96fea62b-6fd4-40b2-8b7d-21b04f456fdd","Type":"ContainerDied","Data":"8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20"} Nov 03 19:22:19.998587 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.998475 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt" event={"ID":"96fea62b-6fd4-40b2-8b7d-21b04f456fdd","Type":"ContainerDied","Data":"f7557d852b052047faae293b91fd04233ea2f319600fa942e749552644418fc6"} Nov 03 19:22:19.998587 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:19.998499 5988 scope.go:122] "RemoveContainer" containerID="8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20" Nov 03 19:22:20.009899 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:20.009878 5988 scope.go:122] "RemoveContainer" containerID="8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20" Nov 03 19:22:20.010241 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:22:20.010218 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20\": container with ID starting with 8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20 not found: ID does not exist" containerID="8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20" Nov 03 19:22:20.010308 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:20.010256 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20"} err="failed to get container status \"8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20\": rpc error: code = NotFound desc = could not find container \"8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20\": container with ID starting with 8f0eb58afc151e66910388d458c9a5d5191fe9bfcf2f15c62832f86acd39fa20 not found: ID does not exist" Nov 03 19:22:20.154784 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:20.154748 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt"] Nov 03 19:22:20.158594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:20.158568 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2e052a-openshift-default-6886cf66dc-p87bt"] Nov 03 19:22:20.598131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:20.598088 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:21.272768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:21.272735 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" path="/var/lib/kubelet/pods/96fea62b-6fd4-40b2-8b7d-21b04f456fdd/volumes" Nov 03 19:22:25.132567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:25.132481 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:25.937741 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:22:25.937703 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:22:29.698557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:29.698522 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:31.812371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:31.812333 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:22:34.213151 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:34.213118 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:38.239669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:38.239633 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:39.690139 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:39.690091 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:39.907880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:39.907837 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:40.104816 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:40.104786 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs"] Nov 03 19:22:40.111113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:40.111084 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:40.150447 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:40.150415 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-6jjrs"] Nov 03 19:22:41.273485 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:41.273437 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4284fd88-cfbc-403c-ae7d-e393d0e82df2" path="/var/lib/kubelet/pods/4284fd88-cfbc-403c-ae7d-e393d0e82df2/volumes" Nov 03 19:22:44.214707 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:44.210619 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:44.631209 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:44.631155 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:46.375886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:46.375851 5988 scope.go:122] "RemoveContainer" containerID="7b5e0230458f463affa347f4d8af293134e73168fe9f8fd99602b18dc1b2276f" Nov 03 19:22:47.704114 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:47.704073 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:47.918044 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:47.918007 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:49.432679 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:49.432643 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:50.048600 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:50.048566 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:50.256896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:50.256859 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:54.321250 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:54.321216 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:55.936646 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:22:55.936608 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:22:56.155724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:56.155673 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:56.412618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:56.412564 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:56.612669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:56.612632 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:58.895398 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:58.895358 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:22:59.971279 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:22:59.971246 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:23:03.491712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:03.491611 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:07.970034 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:07.970001 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:11.876809 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:11.876772 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:12.472573 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:12.472521 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:12.675264 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:12.675218 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:12.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:12.878933 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:17.218589 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:17.218530 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:21.730708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:21.728365 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:21.941828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:21.941791 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:23:25.936885 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:23:25.936845 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:23:26.318646 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:26.318615 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:27.060404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.060362 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn"] Nov 03 19:23:27.092546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.092505 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76ee201-c72zn"] Nov 03 19:23:27.214533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.214496 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4"] Nov 03 19:23:27.215068 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215050 5988 cpu_manager.go:414] "containerMap: removing container" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" Nov 03 19:23:27.215137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215071 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" Nov 03 19:23:27.215137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215103 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f4c8cf53-a227-422a-91c1-61193a15b2e9" containerName="revoke-keys" Nov 03 19:23:27.215137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215112 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c8cf53-a227-422a-91c1-61193a15b2e9" containerName="revoke-keys" Nov 03 19:23:27.215137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215126 5988 cpu_manager.go:414] "containerMap: removing container" podUID="da384f09-39e5-4c50-872c-6cc981339aa0" containerName="maas-api" Nov 03 19:23:27.215137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215134 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da384f09-39e5-4c50-872c-6cc981339aa0" containerName="maas-api" Nov 03 19:23:27.215321 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215308 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fea62b-6fd4-40b2-8b7d-21b04f456fdd" containerName="istio-proxy" Nov 03 19:23:27.215359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215324 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c8cf53-a227-422a-91c1-61193a15b2e9" containerName="revoke-keys" Nov 03 19:23:27.215359 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.215333 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="da384f09-39e5-4c50-872c-6cc981339aa0" containerName="maas-api" Nov 03 19:23:27.218829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.218810 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.258780 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.258747 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.258934 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.258807 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.258985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.258968 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ptl\" (UniqueName: \"kubernetes.io/projected/963119f8-95da-43ea-85c5-65e2d89c2596-kube-api-access-77ptl\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.270597 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.270566 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:23:27.271108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.271073 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:23:27.272862 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.272832 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82576b28-3cd0-4ebc-be44-195e748db46b" path="/var/lib/kubelet/pods/82576b28-3cd0-4ebc-be44-195e748db46b/volumes" Nov 03 19:23:27.273480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.273453 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4"] Nov 03 19:23:27.360345 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.360311 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ptl\" (UniqueName: \"kubernetes.io/projected/963119f8-95da-43ea-85c5-65e2d89c2596-kube-api-access-77ptl\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.360546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.360376 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.360546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.360465 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.361237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.361214 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.362944 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.362923 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.417884 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.417849 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ptl\" (UniqueName: \"kubernetes.io/projected/963119f8-95da-43ea-85c5-65e2d89c2596-kube-api-access-77ptl\") pod \"maas-api-revoke-sub-421c7e4e-hvpp4\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.533037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.533003 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:27.906739 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:27.906715 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4"] Nov 03 19:23:27.910087 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:23:27.910060 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963119f8_95da_43ea_85c5_65e2d89c2596.slice/crio-f9f0f8fc4da7cdbe4f18f05b78f8bd889b8da2f348d1256c1824e9479a7862c8 WatchSource:0}: Error finding container f9f0f8fc4da7cdbe4f18f05b78f8bd889b8da2f348d1256c1824e9479a7862c8: Status 404 returned error can't find the container with id f9f0f8fc4da7cdbe4f18f05b78f8bd889b8da2f348d1256c1824e9479a7862c8 Nov 03 19:23:28.436047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:28.436012 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" event={"ID":"963119f8-95da-43ea-85c5-65e2d89c2596","Type":"ContainerStarted","Data":"07cfb99bae58896051cbd0290e9018f058b8b5070cfe5ff924bfafeb47530b25"} Nov 03 19:23:28.436464 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:28.436067 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" event={"ID":"963119f8-95da-43ea-85c5-65e2d89c2596","Type":"ContainerStarted","Data":"f9f0f8fc4da7cdbe4f18f05b78f8bd889b8da2f348d1256c1824e9479a7862c8"} Nov 03 19:23:28.546334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:28.546266 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" podStartSLOduration=1.546248944 podStartE2EDuration="1.546248944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:28.545173973 +0000 UTC m=+2683.426590076" watchObservedRunningTime="2026-11-03 19:23:28.546248944 +0000 UTC m=+2683.427665047" Nov 03 19:23:29.443564 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:29.443529 5988 generic.go:356] "Generic (PLEG): container finished" podID="963119f8-95da-43ea-85c5-65e2d89c2596" containerID="07cfb99bae58896051cbd0290e9018f058b8b5070cfe5ff924bfafeb47530b25" exitCode=0 Nov 03 19:23:29.444010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:29.443611 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" event={"ID":"963119f8-95da-43ea-85c5-65e2d89c2596","Type":"ContainerDied","Data":"07cfb99bae58896051cbd0290e9018f058b8b5070cfe5ff924bfafeb47530b25"} Nov 03 19:23:30.596071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.596045 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:30.697127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.697098 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963119f8-95da-43ea-85c5-65e2d89c2596-kube-api-access-77ptl\" (UniqueName: \"kubernetes.io/projected/963119f8-95da-43ea-85c5-65e2d89c2596-kube-api-access-77ptl\") pod \"963119f8-95da-43ea-85c5-65e2d89c2596\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " Nov 03 19:23:30.697262 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.697148 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-openshift-ca\") pod \"963119f8-95da-43ea-85c5-65e2d89c2596\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " Nov 03 19:23:30.697326 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.697310 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-xks-ca\") pod \"963119f8-95da-43ea-85c5-65e2d89c2596\" (UID: \"963119f8-95da-43ea-85c5-65e2d89c2596\") " Nov 03 19:23:30.697576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.697552 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-openshift-ca" pod "963119f8-95da-43ea-85c5-65e2d89c2596" (UID: "963119f8-95da-43ea-85c5-65e2d89c2596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:23:30.697761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.697747 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:30.699219 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.699191 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-xks-ca" pod "963119f8-95da-43ea-85c5-65e2d89c2596" (UID: "963119f8-95da-43ea-85c5-65e2d89c2596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:30.699327 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.699258 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963119f8-95da-43ea-85c5-65e2d89c2596-kube-api-access-77ptl" pod "963119f8-95da-43ea-85c5-65e2d89c2596" (UID: "963119f8-95da-43ea-85c5-65e2d89c2596"). InnerVolumeSpecName "kube-api-access-77ptl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:30.798992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.798963 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/963119f8-95da-43ea-85c5-65e2d89c2596-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:30.798992 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.798990 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77ptl\" (UniqueName: \"kubernetes.io/projected/963119f8-95da-43ea-85c5-65e2d89c2596-kube-api-access-77ptl\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:30.917550 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:30.917498 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:31.457596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:31.457512 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" event={"ID":"963119f8-95da-43ea-85c5-65e2d89c2596","Type":"ContainerDied","Data":"f9f0f8fc4da7cdbe4f18f05b78f8bd889b8da2f348d1256c1824e9479a7862c8"} Nov 03 19:23:31.457596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:31.457562 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9f0f8fc4da7cdbe4f18f05b78f8bd889b8da2f348d1256c1824e9479a7862c8" Nov 03 19:23:31.457596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:31.457520 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4" Nov 03 19:23:35.520736 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:35.520699 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:39.408580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:39.408546 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:40.132181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.132149 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg"] Nov 03 19:23:40.132672 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.132658 5988 cpu_manager.go:414] "containerMap: removing container" podUID="963119f8-95da-43ea-85c5-65e2d89c2596" containerName="revoke-keys" Nov 03 19:23:40.132752 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.132673 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963119f8-95da-43ea-85c5-65e2d89c2596" containerName="revoke-keys" Nov 03 19:23:40.132849 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.132838 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="963119f8-95da-43ea-85c5-65e2d89c2596" containerName="revoke-keys" Nov 03 19:23:40.136087 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.136065 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.188500 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.188096 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:23:40.204436 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.204385 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzlm2\" (UniqueName: \"kubernetes.io/projected/e377303f-5708-4b1e-b3a4-43faaebe237f-kube-api-access-gzlm2\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.204712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.204676 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.204922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.204903 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.223428 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.223394 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:40.234108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.234076 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:23:40.237693 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.237661 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg"] Nov 03 19:23:40.306204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.306173 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlm2\" (UniqueName: \"kubernetes.io/projected/e377303f-5708-4b1e-b3a4-43faaebe237f-kube-api-access-gzlm2\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.306371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.306216 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.306371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.306285 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.307091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.307065 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.308643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.308622 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.363590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.363557 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzlm2\" (UniqueName: \"kubernetes.io/projected/e377303f-5708-4b1e-b3a4-43faaebe237f-kube-api-access-gzlm2\") pod \"maas-api-revoke-sub-8600c1f2-2r5hg\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.451482 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.451398 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:40.835194 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:40.835167 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg"] Nov 03 19:23:40.837775 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:23:40.837747 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode377303f_5708_4b1e_b3a4_43faaebe237f.slice/crio-e1e4a1bd44cfba7fd8390fb072b16e09ddfe76b5b55a9b0a267fa1f85c7865f5 WatchSource:0}: Error finding container e1e4a1bd44cfba7fd8390fb072b16e09ddfe76b5b55a9b0a267fa1f85c7865f5: Status 404 returned error can't find the container with id e1e4a1bd44cfba7fd8390fb072b16e09ddfe76b5b55a9b0a267fa1f85c7865f5 Nov 03 19:23:41.526729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:41.526624 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" event={"ID":"e377303f-5708-4b1e-b3a4-43faaebe237f","Type":"ContainerStarted","Data":"9028acb53d27b679a22687fc70f79a364bb066f0ec12711aebdd5b17e4d3a9b3"} Nov 03 19:23:41.526729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:41.526709 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" event={"ID":"e377303f-5708-4b1e-b3a4-43faaebe237f","Type":"ContainerStarted","Data":"e1e4a1bd44cfba7fd8390fb072b16e09ddfe76b5b55a9b0a267fa1f85c7865f5"} Nov 03 19:23:41.636854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:41.636790 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" podStartSLOduration=1.636771701 podStartE2EDuration="1.636771701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:41.634821511 +0000 UTC m=+2696.516237614" watchObservedRunningTime="2026-11-03 19:23:41.636771701 +0000 UTC m=+2696.518187802" Nov 03 19:23:42.533703 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:42.533656 5988 generic.go:356] "Generic (PLEG): container finished" podID="e377303f-5708-4b1e-b3a4-43faaebe237f" containerID="9028acb53d27b679a22687fc70f79a364bb066f0ec12711aebdd5b17e4d3a9b3" exitCode=0 Nov 03 19:23:42.534174 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:42.533717 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" event={"ID":"e377303f-5708-4b1e-b3a4-43faaebe237f","Type":"ContainerDied","Data":"9028acb53d27b679a22687fc70f79a364bb066f0ec12711aebdd5b17e4d3a9b3"} Nov 03 19:23:43.673865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.673837 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:43.752455 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.752425 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-xks-ca\") pod \"e377303f-5708-4b1e-b3a4-43faaebe237f\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " Nov 03 19:23:43.752629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.752618 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e377303f-5708-4b1e-b3a4-43faaebe237f-kube-api-access-gzlm2\" (UniqueName: \"kubernetes.io/projected/e377303f-5708-4b1e-b3a4-43faaebe237f-kube-api-access-gzlm2\") pod \"e377303f-5708-4b1e-b3a4-43faaebe237f\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " Nov 03 19:23:43.752738 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.752717 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-openshift-ca\") pod \"e377303f-5708-4b1e-b3a4-43faaebe237f\" (UID: \"e377303f-5708-4b1e-b3a4-43faaebe237f\") " Nov 03 19:23:43.753094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.753067 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-openshift-ca" pod "e377303f-5708-4b1e-b3a4-43faaebe237f" (UID: "e377303f-5708-4b1e-b3a4-43faaebe237f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:23:43.753199 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.753179 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:43.754610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.754579 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-xks-ca" pod "e377303f-5708-4b1e-b3a4-43faaebe237f" (UID: "e377303f-5708-4b1e-b3a4-43faaebe237f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:23:43.754796 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.754775 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e377303f-5708-4b1e-b3a4-43faaebe237f-kube-api-access-gzlm2" pod "e377303f-5708-4b1e-b3a4-43faaebe237f" (UID: "e377303f-5708-4b1e-b3a4-43faaebe237f"). InnerVolumeSpecName "kube-api-access-gzlm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:23:43.854845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.854812 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzlm2\" (UniqueName: \"kubernetes.io/projected/e377303f-5708-4b1e-b3a4-43faaebe237f-kube-api-access-gzlm2\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:43.854845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:43.854841 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e377303f-5708-4b1e-b3a4-43faaebe237f-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:23:44.547504 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:44.547462 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" event={"ID":"e377303f-5708-4b1e-b3a4-43faaebe237f","Type":"ContainerDied","Data":"e1e4a1bd44cfba7fd8390fb072b16e09ddfe76b5b55a9b0a267fa1f85c7865f5"} Nov 03 19:23:44.547504 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:44.547500 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg" Nov 03 19:23:44.547798 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:44.547514 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1e4a1bd44cfba7fd8390fb072b16e09ddfe76b5b55a9b0a267fa1f85c7865f5" Nov 03 19:23:44.742585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:44.742533 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:48.458463 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:48.458425 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:23:49.293294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:49.293255 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:53.053445 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:53.053407 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv"] Nov 03 19:23:53.107710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:53.107116 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388423c4-vkblv"] Nov 03 19:23:53.272004 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:53.271968 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e5ea48-fa1f-465f-98b4-103fdc69a276" path="/var/lib/kubelet/pods/51e5ea48-fa1f-465f-98b4-103fdc69a276/volumes" Nov 03 19:23:53.901103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:53.901014 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:55.936012 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:23:55.935968 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:23:56.619411 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:56.619378 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:23:58.804890 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:23:58.804847 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:03.316722 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:03.316667 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:05.947486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:05.947448 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:06.407545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.407513 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47"] Nov 03 19:24:06.408047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.408026 5988 cpu_manager.go:414] "containerMap: removing container" podUID="e377303f-5708-4b1e-b3a4-43faaebe237f" containerName="revoke-keys" Nov 03 19:24:06.408047 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.408043 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e377303f-5708-4b1e-b3a4-43faaebe237f" containerName="revoke-keys" Nov 03 19:24:06.408234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.408160 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e377303f-5708-4b1e-b3a4-43faaebe237f" containerName="revoke-keys" Nov 03 19:24:06.412840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.412818 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.464969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.464945 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:24:06.475978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.475952 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:24:06.478218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.478190 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47"] Nov 03 19:24:06.505541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.505517 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.505694 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.505658 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcwk\" (UniqueName: \"kubernetes.io/projected/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-kube-api-access-nrcwk\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.505779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.505760 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.606839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.606810 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.606955 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.606904 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcwk\" (UniqueName: \"kubernetes.io/projected/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-kube-api-access-nrcwk\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.606955 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.606930 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.607457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.607432 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.609225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.609204 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.664217 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.664158 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcwk\" (UniqueName: \"kubernetes.io/projected/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-kube-api-access-nrcwk\") pod \"maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:06.726124 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:06.726099 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:07.096103 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:07.096074 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47"] Nov 03 19:24:07.097713 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:24:07.097661 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939b850d_4cc4_4da4_ad57_7a9b06d9eaf9.slice/crio-f5ef9cfc76a73c6f184993afcac12009afce9f78b00191077397142ba521d710 WatchSource:0}: Error finding container f5ef9cfc76a73c6f184993afcac12009afce9f78b00191077397142ba521d710: Status 404 returned error can't find the container with id f5ef9cfc76a73c6f184993afcac12009afce9f78b00191077397142ba521d710 Nov 03 19:24:07.696094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:07.696011 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" event={"ID":"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9","Type":"ContainerStarted","Data":"578f95ae15274272d04e7e47480d10cee78e7144ebf08577d9bcbff757d81203"} Nov 03 19:24:07.696094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:07.696077 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" event={"ID":"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9","Type":"ContainerStarted","Data":"f5ef9cfc76a73c6f184993afcac12009afce9f78b00191077397142ba521d710"} Nov 03 19:24:07.806395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:07.806343 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" podStartSLOduration=1.806330244 podStartE2EDuration="1.806330244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:07.804177568 +0000 UTC m=+2722.685593669" watchObservedRunningTime="2026-11-03 19:24:07.806330244 +0000 UTC m=+2722.687746346" Nov 03 19:24:08.097244 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.097203 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:08.178909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.178865 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg"] Nov 03 19:24:08.183043 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.183014 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.236740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.236704 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg"] Nov 03 19:24:08.327430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.327397 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.327636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.327485 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vqg\" (UniqueName: \"kubernetes.io/projected/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-kube-api-access-l5vqg\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.327636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.327527 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.429163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.429082 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.429163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.429132 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vqg\" (UniqueName: \"kubernetes.io/projected/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-kube-api-access-l5vqg\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.429163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.429163 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.429897 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.429872 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.431430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.431400 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.486819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.486788 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vqg\" (UniqueName: \"kubernetes.io/projected/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-kube-api-access-l5vqg\") pod \"maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.496625 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.496597 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:08.703039 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.702956 5988 generic.go:356] "Generic (PLEG): container finished" podID="939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" containerID="578f95ae15274272d04e7e47480d10cee78e7144ebf08577d9bcbff757d81203" exitCode=0 Nov 03 19:24:08.703193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.703041 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" event={"ID":"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9","Type":"ContainerDied","Data":"578f95ae15274272d04e7e47480d10cee78e7144ebf08577d9bcbff757d81203"} Nov 03 19:24:08.867098 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:08.867073 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg"] Nov 03 19:24:08.869546 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:24:08.869516 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26df42b5_bf1a_42c9_a0c0_1d2a30aa634d.slice/crio-794d1b6d1a3259594c1d8d40b3475ebb2b1ca4f62f8663eba400cfffcb2d8e0b WatchSource:0}: Error finding container 794d1b6d1a3259594c1d8d40b3475ebb2b1ca4f62f8663eba400cfffcb2d8e0b: Status 404 returned error can't find the container with id 794d1b6d1a3259594c1d8d40b3475ebb2b1ca4f62f8663eba400cfffcb2d8e0b Nov 03 19:24:09.420938 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.420901 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb"] Nov 03 19:24:09.421521 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.421150 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" containerID="cri-o://bb52d4d714127f0c9503747328c0a6b5e4562bb7efe2e4cc8cc0656cb395a6a2" gracePeriod=30 Nov 03 19:24:09.468239 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.468207 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546"] Nov 03 19:24:09.468478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.468450 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" containerID="cri-o://09cd176c85a5c1871facab6f0f4a35d6d561db21fdd014c93a10246bd868dff8" gracePeriod=30 Nov 03 19:24:09.470117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.470091 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls"] Nov 03 19:24:09.470287 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.470262 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" containerID="cri-o://55e5d05cd7e1f5c32c1bedc7fba865c7fed57ef4a3d0d2e0449b33351e9a911a" gracePeriod=30 Nov 03 19:24:09.471953 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.471925 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2"] Nov 03 19:24:09.472127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.472107 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" containerID="cri-o://980cd2e00d5c9b072e11e59555991513bcd07834a0ae0a36d6f9f6b78aff91f0" gracePeriod=30 Nov 03 19:24:09.710459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.710368 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" event={"ID":"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d","Type":"ContainerStarted","Data":"662b881a44a634cecf0043594c449aceedb36f30802c955f6c77aac4b49e3bea"} Nov 03 19:24:09.710459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.710428 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" event={"ID":"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d","Type":"ContainerStarted","Data":"794d1b6d1a3259594c1d8d40b3475ebb2b1ca4f62f8663eba400cfffcb2d8e0b"} Nov 03 19:24:09.851950 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.851921 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:09.944056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.944025 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-openshift-ca\") pod \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " Nov 03 19:24:09.944056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.944061 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-xks-ca\") pod \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " Nov 03 19:24:09.944299 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.944153 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-kube-api-access-nrcwk\" (UniqueName: \"kubernetes.io/projected/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-kube-api-access-nrcwk\") pod \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\" (UID: \"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9\") " Nov 03 19:24:09.944390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.944358 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-openshift-ca" pod "939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" (UID: "939b850d-4cc4-4da4-ad57-7a9b06d9eaf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:09.944606 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.944585 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:09.946263 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.946232 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-xks-ca" pod "939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" (UID: "939b850d-4cc4-4da4-ad57-7a9b06d9eaf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:09.946379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:09.946346 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-kube-api-access-nrcwk" pod "939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" (UID: "939b850d-4cc4-4da4-ad57-7a9b06d9eaf9"). InnerVolumeSpecName "kube-api-access-nrcwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:10.045616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.045529 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:10.045616 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.045564 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrcwk\" (UniqueName: \"kubernetes.io/projected/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9-kube-api-access-nrcwk\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:10.133777 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.133678 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" podStartSLOduration=2.133663716 podStartE2EDuration="2.133663716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:10.133351136 +0000 UTC m=+2725.014767240" watchObservedRunningTime="2026-11-03 19:24:10.133663716 +0000 UTC m=+2725.015079809" Nov 03 19:24:10.717896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.717865 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" Nov 03 19:24:10.718330 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.717887 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47" event={"ID":"939b850d-4cc4-4da4-ad57-7a9b06d9eaf9","Type":"ContainerDied","Data":"f5ef9cfc76a73c6f184993afcac12009afce9f78b00191077397142ba521d710"} Nov 03 19:24:10.718330 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.717948 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5ef9cfc76a73c6f184993afcac12009afce9f78b00191077397142ba521d710" Nov 03 19:24:10.719792 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.719765 5988 generic.go:356] "Generic (PLEG): container finished" podID="26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" containerID="662b881a44a634cecf0043594c449aceedb36f30802c955f6c77aac4b49e3bea" exitCode=0 Nov 03 19:24:10.719912 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:10.719821 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" event={"ID":"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d","Type":"ContainerDied","Data":"662b881a44a634cecf0043594c449aceedb36f30802c955f6c77aac4b49e3bea"} Nov 03 19:24:11.137031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.136993 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x4nc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:24:11.137214 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.137063 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:24:11.709935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.709899 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-2fxrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:24:11.710109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.709956 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:24:11.746983 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.746945 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:24:11.880674 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.880644 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:11.967483 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.967384 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-kube-api-access-l5vqg\" (UniqueName: \"kubernetes.io/projected/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-kube-api-access-l5vqg\") pod \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " Nov 03 19:24:11.967670 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.967530 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-openshift-ca\") pod \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " Nov 03 19:24:11.967764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.967676 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-xks-ca\") pod \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\" (UID: \"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d\") " Nov 03 19:24:11.967947 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.967923 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-openshift-ca" pod "26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" (UID: "26df42b5-bf1a-42c9-a0c0-1d2a30aa634d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:11.968093 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.968078 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:11.969666 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.969645 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-kube-api-access-l5vqg" pod "26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" (UID: "26df42b5-bf1a-42c9-a0c0-1d2a30aa634d"). InnerVolumeSpecName "kube-api-access-l5vqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:11.969767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.969746 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-xks-ca" pod "26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" (UID: "26df42b5-bf1a-42c9-a0c0-1d2a30aa634d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:11.986329 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.986294 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-w6fls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:24:11.986486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:11.986356 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:24:12.069081 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:12.069044 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5vqg\" (UniqueName: \"kubernetes.io/projected/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-kube-api-access-l5vqg\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:12.069081 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:12.069076 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:12.733909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:12.733872 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" Nov 03 19:24:12.734120 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:12.733902 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg" event={"ID":"26df42b5-bf1a-42c9-a0c0-1d2a30aa634d","Type":"ContainerDied","Data":"794d1b6d1a3259594c1d8d40b3475ebb2b1ca4f62f8663eba400cfffcb2d8e0b"} Nov 03 19:24:12.734120 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:12.733966 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="794d1b6d1a3259594c1d8d40b3475ebb2b1ca4f62f8663eba400cfffcb2d8e0b" Nov 03 19:24:12.952656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:12.952586 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:14.754528 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.754490 5988 generic.go:356] "Generic (PLEG): container finished" podID="aa252a91-d63e-439a-a035-605aff921960" containerID="980cd2e00d5c9b072e11e59555991513bcd07834a0ae0a36d6f9f6b78aff91f0" exitCode=0 Nov 03 19:24:14.755009 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.754635 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" event={"ID":"aa252a91-d63e-439a-a035-605aff921960","Type":"ContainerDied","Data":"980cd2e00d5c9b072e11e59555991513bcd07834a0ae0a36d6f9f6b78aff91f0"} Nov 03 19:24:14.756852 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.756822 5988 generic.go:356] "Generic (PLEG): container finished" podID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerID="55e5d05cd7e1f5c32c1bedc7fba865c7fed57ef4a3d0d2e0449b33351e9a911a" exitCode=0 Nov 03 19:24:14.756977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.756908 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" event={"ID":"591c9108-119c-40d3-b301-cd6761ecb0dd","Type":"ContainerDied","Data":"55e5d05cd7e1f5c32c1bedc7fba865c7fed57ef4a3d0d2e0449b33351e9a911a"} Nov 03 19:24:14.759017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.758994 5988 generic.go:356] "Generic (PLEG): container finished" podID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerID="bb52d4d714127f0c9503747328c0a6b5e4562bb7efe2e4cc8cc0656cb395a6a2" exitCode=0 Nov 03 19:24:14.759129 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.759114 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" event={"ID":"4efba1c7-6444-4ccc-94f3-4395a528bcaf","Type":"ContainerDied","Data":"bb52d4d714127f0c9503747328c0a6b5e4562bb7efe2e4cc8cc0656cb395a6a2"} Nov 03 19:24:14.761149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.761127 5988 generic.go:356] "Generic (PLEG): container finished" podID="f40f9779-3386-42a7-973c-db952983f870" containerID="09cd176c85a5c1871facab6f0f4a35d6d561db21fdd014c93a10246bd868dff8" exitCode=0 Nov 03 19:24:14.761258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.761166 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" event={"ID":"f40f9779-3386-42a7-973c-db952983f870","Type":"ContainerDied","Data":"09cd176c85a5c1871facab6f0f4a35d6d561db21fdd014c93a10246bd868dff8"} Nov 03 19:24:14.792972 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.792940 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:24:14.797255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.797229 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:24:14.808769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.808745 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:24:14.812082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.812062 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:24:14.904023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.903987 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-kube-api-access-nqwq6\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-kube-api-access-nqwq6\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.904218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904039 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-token\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-token\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.904218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904065 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-socket\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.904218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904086 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-credential-socket\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.904218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904109 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-certs\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.904218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904126 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-envoy\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.904218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904156 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-credential-socket\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.904218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904191 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-envoy\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904228 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-socket\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904250 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-data\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904282 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-socket\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904311 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa252a91-d63e-439a-a035-605aff921960-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa252a91-d63e-439a-a035-605aff921960-istio-podinfo\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904342 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-data\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904370 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-certs\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904403 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-kube-api-access-n2jfl\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-kube-api-access-n2jfl\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904432 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-socket\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904468 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-envoy\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904504 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-certs\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904539 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-istio-token\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904568 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istiod-ca-cert\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.904612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904593 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-envoy\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904622 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-podinfo\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904669 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-token\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-token\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904760 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-kube-api-access-df488\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-kube-api-access-df488\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904793 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-certs\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904866 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591c9108-119c-40d3-b301-cd6761ecb0dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591c9108-119c-40d3-b301-cd6761ecb0dd-istiod-ca-cert\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904899 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-kube-api-access-v86km\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-kube-api-access-v86km\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904926 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa252a91-d63e-439a-a035-605aff921960-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa252a91-d63e-439a-a035-605aff921960-istiod-ca-cert\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904967 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f40f9779-3386-42a7-973c-db952983f870-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9779-3386-42a7-973c-db952983f870-istio-podinfo\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905032 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-data\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905086 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-credential-socket\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905114 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-data\") pod \"591c9108-119c-40d3-b301-cd6761ecb0dd\" (UID: \"591c9108-119c-40d3-b301-cd6761ecb0dd\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905145 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-credential-socket\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905218 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-podinfo\") pod \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\" (UID: \"4efba1c7-6444-4ccc-94f3-4395a528bcaf\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905262 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40f9779-3386-42a7-973c-db952983f870-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9779-3386-42a7-973c-db952983f870-istiod-ca-cert\") pod \"f40f9779-3386-42a7-973c-db952983f870\" (UID: \"f40f9779-3386-42a7-973c-db952983f870\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904425 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-credential-socket" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905303 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-istio-token\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-istio-token\") pod \"aa252a91-d63e-439a-a035-605aff921960\" (UID: \"aa252a91-d63e-439a-a035-605aff921960\") " Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904432 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-certs" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904444 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-socket" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904661 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-credential-socket" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904744 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-socket" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904743 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-socket" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.904946 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-data" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905129 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-socket" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.905352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905139 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-certs" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905541 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-credential-socket" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905582 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591c9108-119c-40d3-b301-cd6761ecb0dd-istiod-ca-cert" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:14.907055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.905702 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-data" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.906316 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-credential-socket" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.906391 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-data" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907055 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.906418 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-certs" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.907383 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-token" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.907669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.907645 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-certs" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907821 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.907750 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-kube-api-access-n2jfl" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "kube-api-access-n2jfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.907882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.907817 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-envoy" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.907938 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.907891 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-kube-api-access-nqwq6" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "kube-api-access-nqwq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.908015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.907994 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908022 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908074 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908091 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908093 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istiod-ca-cert" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:14.908131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908107 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908123 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908139 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908154 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908170 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908186 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908202 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908219 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2jfl\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-kube-api-access-n2jfl\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908219 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa252a91-d63e-439a-a035-605aff921960-istiod-ca-cert" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908234 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908250 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908266 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908280 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/591c9108-119c-40d3-b301-cd6761ecb0dd-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908294 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908309 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.908406 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908326 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:14.909286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908560 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-data" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.909286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.908645 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40f9779-3386-42a7-973c-db952983f870-istiod-ca-cert" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:14.910149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.910107 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-envoy" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.910430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.910316 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-istio-token" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.910430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.910377 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-envoy" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.910430 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.910421 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-envoy" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:14.910639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.910563 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa252a91-d63e-439a-a035-605aff921960-istio-podinfo" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:14.910756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.910735 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-kube-api-access-df488" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "kube-api-access-df488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.910817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.910741 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-kube-api-access-v86km" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "kube-api-access-v86km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.911090 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.911064 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-podinfo" pod "591c9108-119c-40d3-b301-cd6761ecb0dd" (UID: "591c9108-119c-40d3-b301-cd6761ecb0dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:14.911347 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.911330 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-token" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.911422 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.911409 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-podinfo" pod "4efba1c7-6444-4ccc-94f3-4395a528bcaf" (UID: "4efba1c7-6444-4ccc-94f3-4395a528bcaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:14.911592 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.911573 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-istio-token" pod "aa252a91-d63e-439a-a035-605aff921960" (UID: "aa252a91-d63e-439a-a035-605aff921960"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:14.911774 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:14.911758 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f40f9779-3386-42a7-973c-db952983f870-istio-podinfo" pod "f40f9779-3386-42a7-973c-db952983f870" (UID: "f40f9779-3386-42a7-973c-db952983f870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009896 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40f9779-3386-42a7-973c-db952983f870-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009928 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009940 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqwq6\" (UniqueName: \"kubernetes.io/projected/aa252a91-d63e-439a-a035-605aff921960-kube-api-access-nqwq6\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009950 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40f9779-3386-42a7-973c-db952983f870-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009961 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa252a91-d63e-439a-a035-605aff921960-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009969 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009977 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40f9779-3386-42a7-973c-db952983f870-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009986 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.009985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.009994 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010003 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/591c9108-119c-40d3-b301-cd6761ecb0dd-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010011 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010019 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df488\" (UniqueName: \"kubernetes.io/projected/591c9108-119c-40d3-b301-cd6761ecb0dd-kube-api-access-df488\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010028 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86km\" (UniqueName: \"kubernetes.io/projected/4efba1c7-6444-4ccc-94f3-4395a528bcaf-kube-api-access-v86km\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010036 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa252a91-d63e-439a-a035-605aff921960-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010045 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40f9779-3386-42a7-973c-db952983f870-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010053 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa252a91-d63e-439a-a035-605aff921960-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.010391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.010062 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efba1c7-6444-4ccc-94f3-4395a528bcaf-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:15.768621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.768592 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" Nov 03 19:24:15.769091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.768637 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546" event={"ID":"f40f9779-3386-42a7-973c-db952983f870","Type":"ContainerDied","Data":"3a1537c8a2bfc31c73abd7b55bdd7a6d9def18aa8d1cabc710ec82f6ae841c91"} Nov 03 19:24:15.769091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.768720 5988 scope.go:122] "RemoveContainer" containerID="09cd176c85a5c1871facab6f0f4a35d6d561db21fdd014c93a10246bd868dff8" Nov 03 19:24:15.770171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.770155 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" Nov 03 19:24:15.770262 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.770208 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2" event={"ID":"aa252a91-d63e-439a-a035-605aff921960","Type":"ContainerDied","Data":"3ccc58f424035093771e083c19f56d5d6035afbc5d8cb180f77bee37ce70e8bd"} Nov 03 19:24:15.771673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.771657 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" Nov 03 19:24:15.771833 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.771705 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls" event={"ID":"591c9108-119c-40d3-b301-cd6761ecb0dd","Type":"ContainerDied","Data":"3d54a489c822092864ffb29f1f1fd76cd184e367d16a5ec06236cf6134558fd5"} Nov 03 19:24:15.779171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.778951 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" event={"ID":"4efba1c7-6444-4ccc-94f3-4395a528bcaf","Type":"ContainerDied","Data":"b15e73af9b0e0f620ba50dfb4916ea03e4e31be2792e7d8523bca661c3f8b2bb"} Nov 03 19:24:15.779171 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.779085 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb" Nov 03 19:24:15.782544 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.782524 5988 scope.go:122] "RemoveContainer" containerID="980cd2e00d5c9b072e11e59555991513bcd07834a0ae0a36d6f9f6b78aff91f0" Nov 03 19:24:15.791832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.791793 5988 scope.go:122] "RemoveContainer" containerID="55e5d05cd7e1f5c32c1bedc7fba865c7fed57ef4a3d0d2e0449b33351e9a911a" Nov 03 19:24:15.801053 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.801032 5988 scope.go:122] "RemoveContainer" containerID="bb52d4d714127f0c9503747328c0a6b5e4562bb7efe2e4cc8cc0656cb395a6a2" Nov 03 19:24:15.927114 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.927077 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb"] Nov 03 19:24:15.973845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:15.973802 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-2fxrb"] Nov 03 19:24:16.038403 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:16.038297 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2"] Nov 03 19:24:16.040237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:16.040211 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x4nc2"] Nov 03 19:24:16.145428 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:16.145391 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls"] Nov 03 19:24:16.151825 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:16.151789 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-w6fls"] Nov 03 19:24:16.255391 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:16.255354 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546"] Nov 03 19:24:16.259995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:16.259967 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hm546"] Nov 03 19:24:17.272206 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:17.272168 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" path="/var/lib/kubelet/pods/4efba1c7-6444-4ccc-94f3-4395a528bcaf/volumes" Nov 03 19:24:17.272847 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:17.272826 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" path="/var/lib/kubelet/pods/591c9108-119c-40d3-b301-cd6761ecb0dd/volumes" Nov 03 19:24:17.273312 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:17.273293 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa252a91-d63e-439a-a035-605aff921960" path="/var/lib/kubelet/pods/aa252a91-d63e-439a-a035-605aff921960/volumes" Nov 03 19:24:17.274065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:17.274045 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40f9779-3386-42a7-973c-db952983f870" path="/var/lib/kubelet/pods/f40f9779-3386-42a7-973c-db952983f870/volumes" Nov 03 19:24:17.608011 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:17.607954 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:17.937525 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:17.937421 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s"] Nov 03 19:24:18.047935 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:18.047884 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m"] Nov 03 19:24:18.093919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:18.093883 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t"] Nov 03 19:24:18.095822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:18.095797 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d117c998-zvr5m"] Nov 03 19:24:18.402413 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:18.402372 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:18.606120 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:18.606079 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:19.045752 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:19.045721 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl"] Nov 03 19:24:19.048478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:19.048450 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1bf6db0-gvddl"] Nov 03 19:24:19.273143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:19.273112 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c2e453-db67-4b49-9474-03272e83175b" path="/var/lib/kubelet/pods/a6c2e453-db67-4b49-9474-03272e83175b/volumes" Nov 03 19:24:19.273541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:19.273528 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7" path="/var/lib/kubelet/pods/f178a2bb-b5b0-43d5-b6e2-85d5fa9bdbb7/volumes" Nov 03 19:24:20.991035 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:20.991000 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:21.197817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:21.197784 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:22.209495 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:22.209450 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:22.938678 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:22.938623 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" podUID="e56dd326-b071-4d84-8d5a-352173b3504d" containerName="payload-processing" containerID="cri-o://0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3" gracePeriod=25 Nov 03 19:24:23.094294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.094255 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" podUID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" containerName="payload-pre-processing" containerID="cri-o://ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e" gracePeriod=25 Nov 03 19:24:23.193402 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.193340 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:24:23.296603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.296570 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56dd326-b071-4d84-8d5a-352173b3504d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56dd326-b071-4d84-8d5a-352173b3504d-plugins-config-volume\") pod \"e56dd326-b071-4d84-8d5a-352173b3504d\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " Nov 03 19:24:23.296946 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.296630 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56dd326-b071-4d84-8d5a-352173b3504d-kube-api-access-w9hpz\" (UniqueName: \"kubernetes.io/projected/e56dd326-b071-4d84-8d5a-352173b3504d-kube-api-access-w9hpz\") pod \"e56dd326-b071-4d84-8d5a-352173b3504d\" (UID: \"e56dd326-b071-4d84-8d5a-352173b3504d\") " Nov 03 19:24:23.297000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.296935 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56dd326-b071-4d84-8d5a-352173b3504d-plugins-config-volume" pod "e56dd326-b071-4d84-8d5a-352173b3504d" (UID: "e56dd326-b071-4d84-8d5a-352173b3504d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:23.298674 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.298650 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56dd326-b071-4d84-8d5a-352173b3504d-kube-api-access-w9hpz" pod "e56dd326-b071-4d84-8d5a-352173b3504d" (UID: "e56dd326-b071-4d84-8d5a-352173b3504d"). InnerVolumeSpecName "kube-api-access-w9hpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:23.398294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.398269 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56dd326-b071-4d84-8d5a-352173b3504d-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:23.398294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.398293 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9hpz\" (UniqueName: \"kubernetes.io/projected/e56dd326-b071-4d84-8d5a-352173b3504d-kube-api-access-w9hpz\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:23.833666 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.833630 5988 generic.go:356] "Generic (PLEG): container finished" podID="e56dd326-b071-4d84-8d5a-352173b3504d" containerID="0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3" exitCode=0 Nov 03 19:24:23.833855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.833715 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" Nov 03 19:24:23.833855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.833723 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" event={"ID":"e56dd326-b071-4d84-8d5a-352173b3504d","Type":"ContainerDied","Data":"0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3"} Nov 03 19:24:23.833855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.833854 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s" event={"ID":"e56dd326-b071-4d84-8d5a-352173b3504d","Type":"ContainerDied","Data":"0fa878841e89f44cf7b919bb0c8b34d91dbe88ec8d86b5abab7fab8158373550"} Nov 03 19:24:23.833981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.833885 5988 scope.go:122] "RemoveContainer" containerID="0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3" Nov 03 19:24:23.844030 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.844010 5988 scope.go:122] "RemoveContainer" containerID="0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3" Nov 03 19:24:23.844574 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:24:23.844554 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3\": container with ID starting with 0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3 not found: ID does not exist" containerID="0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3" Nov 03 19:24:23.844659 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.844580 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3"} err="failed to get container status \"0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3\": rpc error: code = NotFound desc = could not find container \"0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3\": container with ID starting with 0ccf4187c6b842cc4c668a3062daad6f27c601e31fc3dd4fe7cd0aa80a9525f3 not found: ID does not exist" Nov 03 19:24:23.988029 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:23.987997 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s"] Nov 03 19:24:24.034388 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:24.034356 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9dd9f2-674c89d7cc-s977s"] Nov 03 19:24:25.272835 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:25.272799 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e56dd326-b071-4d84-8d5a-352173b3504d" path="/var/lib/kubelet/pods/e56dd326-b071-4d84-8d5a-352173b3504d/volumes" Nov 03 19:24:25.936823 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:24:25.936782 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:24:26.207052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:26.206930 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:27.677426 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:27.677391 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:27.906881 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:27.906850 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:28.109718 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:28.109671 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:28.216955 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:28.216921 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg"] Nov 03 19:24:28.263333 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:28.263300 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d74f801-tp4zg"] Nov 03 19:24:28.705599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:28.705565 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:29.272362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:29.272328 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" path="/var/lib/kubelet/pods/26df42b5-bf1a-42c9-a0c0-1d2a30aa634d/volumes" Nov 03 19:24:29.831230 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:29.831188 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" podUID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:24:32.703206 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:32.703176 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:33.604349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:33.604315 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl"] Nov 03 19:24:33.738839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:33.738806 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:34.806363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:34.806330 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:24:37.309765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:37.309728 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:39.830524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:39.830481 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" podUID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 19:24:41.340305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:41.340273 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:42.785919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:42.785884 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:43.034003 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.033971 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:43.187159 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.187122 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47"] Nov 03 19:24:43.233484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.233454 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9dd9f2-tjm47"] Nov 03 19:24:43.249267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.249228 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:43.275215 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.275178 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" path="/var/lib/kubelet/pods/939b850d-4cc4-4da4-ad57-7a9b06d9eaf9/volumes" Nov 03 19:24:43.346554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.346523 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:24:43.507312 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.507218 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5f3c45-1e8f-4e40-862a-73f55de550ea-kube-api-access-dt25c\" (UniqueName: \"kubernetes.io/projected/ba5f3c45-1e8f-4e40-862a-73f55de550ea-kube-api-access-dt25c\") pod \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " Nov 03 19:24:43.507471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.507353 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5f3c45-1e8f-4e40-862a-73f55de550ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5f3c45-1e8f-4e40-862a-73f55de550ea-plugins-config-volume\") pod \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\" (UID: \"ba5f3c45-1e8f-4e40-862a-73f55de550ea\") " Nov 03 19:24:43.507775 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.507753 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5f3c45-1e8f-4e40-862a-73f55de550ea-plugins-config-volume" pod "ba5f3c45-1e8f-4e40-862a-73f55de550ea" (UID: "ba5f3c45-1e8f-4e40-862a-73f55de550ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:43.509529 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.509508 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5f3c45-1e8f-4e40-862a-73f55de550ea-kube-api-access-dt25c" pod "ba5f3c45-1e8f-4e40-862a-73f55de550ea" (UID: "ba5f3c45-1e8f-4e40-862a-73f55de550ea"). InnerVolumeSpecName "kube-api-access-dt25c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:43.609525 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.609488 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba5f3c45-1e8f-4e40-862a-73f55de550ea-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:43.609525 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.609517 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt25c\" (UniqueName: \"kubernetes.io/projected/ba5f3c45-1e8f-4e40-862a-73f55de550ea-kube-api-access-dt25c\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:43.724139 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.724099 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:43.955758 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.955720 5988 generic.go:356] "Generic (PLEG): container finished" podID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" containerID="ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e" exitCode=0 Nov 03 19:24:43.956179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.955772 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" Nov 03 19:24:43.956179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.955802 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" event={"ID":"ba5f3c45-1e8f-4e40-862a-73f55de550ea","Type":"ContainerDied","Data":"ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e"} Nov 03 19:24:43.956179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.955861 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t" event={"ID":"ba5f3c45-1e8f-4e40-862a-73f55de550ea","Type":"ContainerDied","Data":"f1a155d1960054cd796840e24dd127a2b541c0b6da2e653d8e22f82b6462ed0b"} Nov 03 19:24:43.956179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.955888 5988 scope.go:122] "RemoveContainer" containerID="ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e" Nov 03 19:24:43.966783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.966757 5988 scope.go:122] "RemoveContainer" containerID="ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e" Nov 03 19:24:43.967081 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:24:43.967058 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e\": container with ID starting with ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e not found: ID does not exist" containerID="ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e" Nov 03 19:24:43.967134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:43.967092 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e"} err="failed to get container status \"ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e\": rpc error: code = NotFound desc = could not find container \"ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e\": container with ID starting with ea85d026e1c3a076efd639f681b14ff0d6d914f21c448860c9b17c139707544e not found: ID does not exist" Nov 03 19:24:44.109451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:44.109416 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t"] Nov 03 19:24:44.156121 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:44.156082 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9dd9f2-58b64d8db7-7x68t"] Nov 03 19:24:45.272983 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:45.272950 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" path="/var/lib/kubelet/pods/ba5f3c45-1e8f-4e40-862a-73f55de550ea/volumes" Nov 03 19:24:46.421297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:46.421267 5988 scope.go:122] "RemoveContainer" containerID="9f3fc9e048460c54b385b6d47805cefd0e99f082b3c160786f0082c55f8a3dd4" Nov 03 19:24:46.430590 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:46.430570 5988 scope.go:122] "RemoveContainer" containerID="7384142c3174f9ca6d8fd2d809316bea44b20069f06bb4863a0f690adf98885f" Nov 03 19:24:46.438974 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:46.438954 5988 scope.go:122] "RemoveContainer" containerID="cabab46b3e4cc218c61efdb529d443d5d317fbe3ef9923fa73359ea766450eef" Nov 03 19:24:47.817660 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:47.817614 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:48.624636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.624587 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" podUID="858e40ab-0455-4e17-bc4d-ff6163b3e750" containerName="main" containerID="cri-o://5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed" gracePeriod=45 Nov 03 19:24:48.869885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.869855 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:24:48.973524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973433 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-home\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-home\") pod \"858e40ab-0455-4e17-bc4d-ff6163b3e750\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " Nov 03 19:24:48.973524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973509 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-model-cache\") pod \"858e40ab-0455-4e17-bc4d-ff6163b3e750\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " Nov 03 19:24:48.973793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973565 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858e40ab-0455-4e17-bc4d-ff6163b3e750-kube-api-access-lfhxn\" (UniqueName: \"kubernetes.io/projected/858e40ab-0455-4e17-bc4d-ff6163b3e750-kube-api-access-lfhxn\") pod \"858e40ab-0455-4e17-bc4d-ff6163b3e750\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " Nov 03 19:24:48.973793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973619 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/858e40ab-0455-4e17-bc4d-ff6163b3e750-tls-certs\" (UniqueName: \"kubernetes.io/secret/858e40ab-0455-4e17-bc4d-ff6163b3e750-tls-certs\") pod \"858e40ab-0455-4e17-bc4d-ff6163b3e750\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " Nov 03 19:24:48.973793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973714 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-tmp-dir\") pod \"858e40ab-0455-4e17-bc4d-ff6163b3e750\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " Nov 03 19:24:48.973793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973780 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-dshm\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-dshm\") pod \"858e40ab-0455-4e17-bc4d-ff6163b3e750\" (UID: \"858e40ab-0455-4e17-bc4d-ff6163b3e750\") " Nov 03 19:24:48.974026 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973765 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-home" pod "858e40ab-0455-4e17-bc4d-ff6163b3e750" (UID: "858e40ab-0455-4e17-bc4d-ff6163b3e750"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:48.974026 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.973838 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-model-cache" pod "858e40ab-0455-4e17-bc4d-ff6163b3e750" (UID: "858e40ab-0455-4e17-bc4d-ff6163b3e750"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:48.974131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.974032 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-tmp-dir" pod "858e40ab-0455-4e17-bc4d-ff6163b3e750" (UID: "858e40ab-0455-4e17-bc4d-ff6163b3e750"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:48.974202 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.974183 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:48.974240 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.974207 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:48.974240 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.974221 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:48.975959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.975938 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858e40ab-0455-4e17-bc4d-ff6163b3e750-kube-api-access-lfhxn" pod "858e40ab-0455-4e17-bc4d-ff6163b3e750" (UID: "858e40ab-0455-4e17-bc4d-ff6163b3e750"). InnerVolumeSpecName "kube-api-access-lfhxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:48.976048 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.975970 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858e40ab-0455-4e17-bc4d-ff6163b3e750-tls-certs" pod "858e40ab-0455-4e17-bc4d-ff6163b3e750" (UID: "858e40ab-0455-4e17-bc4d-ff6163b3e750"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:48.976048 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.976002 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-dshm" pod "858e40ab-0455-4e17-bc4d-ff6163b3e750" (UID: "858e40ab-0455-4e17-bc4d-ff6163b3e750"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:48.993329 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.993293 5988 generic.go:356] "Generic (PLEG): container finished" podID="858e40ab-0455-4e17-bc4d-ff6163b3e750" containerID="5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed" exitCode=0 Nov 03 19:24:48.993487 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.993468 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" event={"ID":"858e40ab-0455-4e17-bc4d-ff6163b3e750","Type":"ContainerDied","Data":"5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed"} Nov 03 19:24:48.993545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.993507 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" event={"ID":"858e40ab-0455-4e17-bc4d-ff6163b3e750","Type":"ContainerDied","Data":"e2de5494b44c8de8ff5ea8842b4ff58f578a96768e3c7a1aeaea276dc80f0db2"} Nov 03 19:24:48.993545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.993526 5988 scope.go:122] "RemoveContainer" containerID="5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed" Nov 03 19:24:48.993656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:48.993646 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl" Nov 03 19:24:49.005562 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.005543 5988 scope.go:122] "RemoveContainer" containerID="5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed" Nov 03 19:24:49.005911 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:24:49.005885 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed\": container with ID starting with 5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed not found: ID does not exist" containerID="5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed" Nov 03 19:24:49.006013 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.005919 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed"} err="failed to get container status \"5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed\": rpc error: code = NotFound desc = could not find container \"5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed\": container with ID starting with 5352812ea746a1859bf68c81a30710467250cce2e1d2e66b96724fedd1ef9fed not found: ID does not exist" Nov 03 19:24:49.075264 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.075219 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhxn\" (UniqueName: \"kubernetes.io/projected/858e40ab-0455-4e17-bc4d-ff6163b3e750-kube-api-access-lfhxn\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:49.075264 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.075256 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/858e40ab-0455-4e17-bc4d-ff6163b3e750-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:49.075264 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.075265 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/858e40ab-0455-4e17-bc4d-ff6163b3e750-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:49.149139 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.149102 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl"] Nov 03 19:24:49.155117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.155091 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3d74f801/e2e-lifecycle-model-3d74f801-kserve-575666fddf-k8csl"] Nov 03 19:24:49.272661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.272579 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858e40ab-0455-4e17-bc4d-ff6163b3e750" path="/var/lib/kubelet/pods/858e40ab-0455-4e17-bc4d-ff6163b3e750/volumes" Nov 03 19:24:49.901291 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:49.901259 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:50.146049 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:50.145984 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:50.242576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:50.242492 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl"] Nov 03 19:24:50.242768 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:50.242714 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" containerID="cri-o://3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407" gracePeriod=30 Nov 03 19:24:50.436545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:50.436494 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:51.045624 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.045592 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk"] Nov 03 19:24:51.046158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046124 5988 cpu_manager.go:414] "containerMap: removing container" podUID="858e40ab-0455-4e17-bc4d-ff6163b3e750" containerName="main" Nov 03 19:24:51.046158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046146 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858e40ab-0455-4e17-bc4d-ff6163b3e750" containerName="main" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046167 5988 cpu_manager.go:414] "containerMap: removing container" podUID="939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" containerName="revoke-keys" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046175 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" containerName="revoke-keys" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046197 5988 cpu_manager.go:414] "containerMap: removing container" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046206 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046221 5988 cpu_manager.go:414] "containerMap: removing container" podUID="26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" containerName="revoke-keys" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046229 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" containerName="revoke-keys" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046242 5988 cpu_manager.go:414] "containerMap: removing container" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046247 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046256 5988 cpu_manager.go:414] "containerMap: removing container" podUID="e56dd326-b071-4d84-8d5a-352173b3504d" containerName="payload-processing" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046261 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56dd326-b071-4d84-8d5a-352173b3504d" containerName="payload-processing" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046287 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" Nov 03 19:24:51.046286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046293 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046305 5988 cpu_manager.go:414] "containerMap: removing container" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046310 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046317 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" containerName="payload-pre-processing" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046321 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" containerName="payload-pre-processing" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046396 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40f9779-3386-42a7-973c-db952983f870" containerName="istio-proxy" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046405 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="591c9108-119c-40d3-b301-cd6761ecb0dd" containerName="istio-proxy" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046411 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5f3c45-1e8f-4e40-862a-73f55de550ea" containerName="payload-pre-processing" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046418 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="858e40ab-0455-4e17-bc4d-ff6163b3e750" containerName="main" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046426 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="939b850d-4cc4-4da4-ad57-7a9b06d9eaf9" containerName="revoke-keys" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046432 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa252a91-d63e-439a-a035-605aff921960" containerName="istio-proxy" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046439 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="26df42b5-bf1a-42c9-a0c0-1d2a30aa634d" containerName="revoke-keys" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046445 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efba1c7-6444-4ccc-94f3-4395a528bcaf" containerName="istio-proxy" Nov 03 19:24:51.046766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.046450 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56dd326-b071-4d84-8d5a-352173b3504d" containerName="payload-processing" Nov 03 19:24:51.050969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.050945 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.091730 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.091678 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.091848 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.091782 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.091891 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.091845 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hx6j\" (UniqueName: \"kubernetes.io/projected/1be49e3f-a674-48b4-9d69-cb5db05fe570-kube-api-access-7hx6j\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.103794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.103770 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk"] Nov 03 19:24:51.148202 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.148173 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:24:51.148349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.148215 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:24:51.192344 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.192317 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.192478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.192398 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hx6j\" (UniqueName: \"kubernetes.io/projected/1be49e3f-a674-48b4-9d69-cb5db05fe570-kube-api-access-7hx6j\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.192478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.192450 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.193107 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.193088 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.194669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.194650 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.249924 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.249895 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hx6j\" (UniqueName: \"kubernetes.io/projected/1be49e3f-a674-48b4-9d69-cb5db05fe570-kube-api-access-7hx6j\") pod \"maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.364283 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.364263 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:51.733015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:51.732984 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk"] Nov 03 19:24:51.734875 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:24:51.734849 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be49e3f_a674_48b4_9d69_cb5db05fe570.slice/crio-4576fd04deae13229aa7ab53e8c65ef72635e833148a313979578ab6a1689b1c WatchSource:0}: Error finding container 4576fd04deae13229aa7ab53e8c65ef72635e833148a313979578ab6a1689b1c: Status 404 returned error can't find the container with id 4576fd04deae13229aa7ab53e8c65ef72635e833148a313979578ab6a1689b1c Nov 03 19:24:52.022990 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:52.022930 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" event={"ID":"1be49e3f-a674-48b4-9d69-cb5db05fe570","Type":"ContainerStarted","Data":"4576fd04deae13229aa7ab53e8c65ef72635e833148a313979578ab6a1689b1c"} Nov 03 19:24:52.524503 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:52.524465 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:53.031315 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:53.031278 5988 generic.go:356] "Generic (PLEG): container finished" podID="1be49e3f-a674-48b4-9d69-cb5db05fe570" containerID="d45555f77bb871287fde07953b3a9f3405478fd518d43cc3a50a768e4f5ac15d" exitCode=0 Nov 03 19:24:53.031501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:53.031341 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" event={"ID":"1be49e3f-a674-48b4-9d69-cb5db05fe570","Type":"ContainerDied","Data":"d45555f77bb871287fde07953b3a9f3405478fd518d43cc3a50a768e4f5ac15d"} Nov 03 19:24:54.180305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.180280 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:54.221134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.221101 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-openshift-ca\") pod \"1be49e3f-a674-48b4-9d69-cb5db05fe570\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " Nov 03 19:24:54.221323 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.221304 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-xks-ca\") pod \"1be49e3f-a674-48b4-9d69-cb5db05fe570\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " Nov 03 19:24:54.221420 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.221404 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be49e3f-a674-48b4-9d69-cb5db05fe570-kube-api-access-7hx6j\" (UniqueName: \"kubernetes.io/projected/1be49e3f-a674-48b4-9d69-cb5db05fe570-kube-api-access-7hx6j\") pod \"1be49e3f-a674-48b4-9d69-cb5db05fe570\" (UID: \"1be49e3f-a674-48b4-9d69-cb5db05fe570\") " Nov 03 19:24:54.221535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.221514 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-openshift-ca" pod "1be49e3f-a674-48b4-9d69-cb5db05fe570" (UID: "1be49e3f-a674-48b4-9d69-cb5db05fe570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:54.221751 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.221734 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:54.223386 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.223367 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-xks-ca" pod "1be49e3f-a674-48b4-9d69-cb5db05fe570" (UID: "1be49e3f-a674-48b4-9d69-cb5db05fe570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:24:54.223522 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.223499 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be49e3f-a674-48b4-9d69-cb5db05fe570-kube-api-access-7hx6j" pod "1be49e3f-a674-48b4-9d69-cb5db05fe570" (UID: "1be49e3f-a674-48b4-9d69-cb5db05fe570"). InnerVolumeSpecName "kube-api-access-7hx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:54.323340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.323308 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1be49e3f-a674-48b4-9d69-cb5db05fe570-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:54.323340 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:54.323341 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hx6j\" (UniqueName: \"kubernetes.io/projected/1be49e3f-a674-48b4-9d69-cb5db05fe570-kube-api-access-7hx6j\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.042981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.042942 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" event={"ID":"1be49e3f-a674-48b4-9d69-cb5db05fe570","Type":"ContainerDied","Data":"4576fd04deae13229aa7ab53e8c65ef72635e833148a313979578ab6a1689b1c"} Nov 03 19:24:55.042981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.042977 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk" Nov 03 19:24:55.043165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.043006 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4576fd04deae13229aa7ab53e8c65ef72635e833148a313979578ab6a1689b1c" Nov 03 19:24:55.501104 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.501078 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:24:55.536149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536116 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istiod-ca-cert\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536193 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-data\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536221 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-kube-api-access-5tmkr\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-kube-api-access-5tmkr\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536261 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-podinfo\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536423 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536330 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-certs\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536423 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536363 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-token\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-token\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536423 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536405 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-envoy\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536452 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-socket\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536498 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-credential-socket\") pod \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\" (UID: \"807178f7-57ea-44bc-94ed-56e4f3ea3cae\") " Nov 03 19:24:55.536585 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536517 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-data" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:55.536776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536634 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-certs" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:55.536776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536705 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istiod-ca-cert" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:24:55.536863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536778 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-socket" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:55.536905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536868 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-credential-socket" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:55.536991 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536969 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.537049 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.536994 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.537049 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.537007 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.537049 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.537021 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.537049 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.537036 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.538574 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.538544 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-podinfo" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:24:55.538725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.538576 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-envoy" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:24:55.538725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.538655 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-kube-api-access-5tmkr" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "kube-api-access-5tmkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:55.538725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.538711 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-token" pod "807178f7-57ea-44bc-94ed-56e4f3ea3cae" (UID: "807178f7-57ea-44bc-94ed-56e4f3ea3cae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:24:55.645875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.645776 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.645875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.645811 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmkr\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-kube-api-access-5tmkr\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.645875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.645824 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.645875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:55.645839 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/807178f7-57ea-44bc-94ed-56e4f3ea3cae-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:24:55.937154 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:24:55.937062 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:24:56.050029 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.049993 5988 generic.go:356] "Generic (PLEG): container finished" podID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerID="3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407" exitCode=0 Nov 03 19:24:56.050230 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.050053 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" Nov 03 19:24:56.050230 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.050076 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" event={"ID":"807178f7-57ea-44bc-94ed-56e4f3ea3cae","Type":"ContainerDied","Data":"3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407"} Nov 03 19:24:56.050230 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.050157 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl" event={"ID":"807178f7-57ea-44bc-94ed-56e4f3ea3cae","Type":"ContainerDied","Data":"aa96bd7bd9dfc21a6c95e865785282297b31a8688ad10c17357c6fb93b7da150"} Nov 03 19:24:56.050230 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.050182 5988 scope.go:122] "RemoveContainer" containerID="3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407" Nov 03 19:24:56.060918 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.060901 5988 scope.go:122] "RemoveContainer" containerID="49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f" Nov 03 19:24:56.069316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.069300 5988 scope.go:122] "RemoveContainer" containerID="3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407" Nov 03 19:24:56.069576 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:24:56.069556 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407\": container with ID starting with 3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407 not found: ID does not exist" containerID="3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407" Nov 03 19:24:56.069617 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.069585 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407"} err="failed to get container status \"3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407\": rpc error: code = NotFound desc = could not find container \"3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407\": container with ID starting with 3948eb7e253a85426eabe52a9edf2a3d4fe479c3e3f5a2e5eb6c0a9cf7784407 not found: ID does not exist" Nov 03 19:24:56.069617 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.069604 5988 scope.go:122] "RemoveContainer" containerID="49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f" Nov 03 19:24:56.069883 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:24:56.069866 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f\": container with ID starting with 49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f not found: ID does not exist" containerID="49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f" Nov 03 19:24:56.069936 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.069887 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f"} err="failed to get container status \"49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f\": rpc error: code = NotFound desc = could not find container \"49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f\": container with ID starting with 49615c979bf0b0e36c375f5a49af4599789b510e32d6a1c45eacf2c93161896f not found: ID does not exist" Nov 03 19:24:56.207730 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.207640 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl"] Nov 03 19:24:56.253867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.253834 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9dd9f2-openshift-default-db6f4568c-k2wkl"] Nov 03 19:24:56.966056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:56.966024 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:57.168093 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:57.168047 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:24:57.272211 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:57.272143 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" path="/var/lib/kubelet/pods/807178f7-57ea-44bc-94ed-56e4f3ea3cae/volumes" Nov 03 19:24:59.847597 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:24:59.847561 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:25:01.626350 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:01.626316 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:02.570618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:02.570578 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4"] Nov 03 19:25:03.010640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:03.010549 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:03.328319 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:03.328275 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:06.247555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:06.247519 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:07.571566 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.571498 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" podUID="a0fb572c-57ca-401c-8222-37dfb13aa9f7" containerName="payload-processing" containerID="cri-o://2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e" gracePeriod=25 Nov 03 19:25:07.822921 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.822845 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:25:07.861793 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.861761 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fb572c-57ca-401c-8222-37dfb13aa9f7-kube-api-access-rkmmm\" (UniqueName: \"kubernetes.io/projected/a0fb572c-57ca-401c-8222-37dfb13aa9f7-kube-api-access-rkmmm\") pod \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " Nov 03 19:25:07.861936 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.861921 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fb572c-57ca-401c-8222-37dfb13aa9f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fb572c-57ca-401c-8222-37dfb13aa9f7-plugins-config-volume\") pod \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\" (UID: \"a0fb572c-57ca-401c-8222-37dfb13aa9f7\") " Nov 03 19:25:07.862286 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.862250 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fb572c-57ca-401c-8222-37dfb13aa9f7-plugins-config-volume" pod "a0fb572c-57ca-401c-8222-37dfb13aa9f7" (UID: "a0fb572c-57ca-401c-8222-37dfb13aa9f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:07.863825 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.863797 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fb572c-57ca-401c-8222-37dfb13aa9f7-kube-api-access-rkmmm" pod "a0fb572c-57ca-401c-8222-37dfb13aa9f7" (UID: "a0fb572c-57ca-401c-8222-37dfb13aa9f7"). InnerVolumeSpecName "kube-api-access-rkmmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:07.963237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.963192 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0fb572c-57ca-401c-8222-37dfb13aa9f7-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:07.963237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:07.963229 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmmm\" (UniqueName: \"kubernetes.io/projected/a0fb572c-57ca-401c-8222-37dfb13aa9f7-kube-api-access-rkmmm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:08.126439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.126352 5988 generic.go:356] "Generic (PLEG): container finished" podID="a0fb572c-57ca-401c-8222-37dfb13aa9f7" containerID="2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e" exitCode=0 Nov 03 19:25:08.126439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.126406 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" Nov 03 19:25:08.126706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.126436 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" event={"ID":"a0fb572c-57ca-401c-8222-37dfb13aa9f7","Type":"ContainerDied","Data":"2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e"} Nov 03 19:25:08.126706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.126504 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4" event={"ID":"a0fb572c-57ca-401c-8222-37dfb13aa9f7","Type":"ContainerDied","Data":"c147ccf1208a7233d144bd06bb555ab6b5817b4fefbf4fa7e1285ba1ac590496"} Nov 03 19:25:08.126706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.126534 5988 scope.go:122] "RemoveContainer" containerID="2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e" Nov 03 19:25:08.136633 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.136608 5988 scope.go:122] "RemoveContainer" containerID="2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e" Nov 03 19:25:08.136917 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:25:08.136897 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e\": container with ID starting with 2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e not found: ID does not exist" containerID="2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e" Nov 03 19:25:08.136988 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.136929 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e"} err="failed to get container status \"2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e\": rpc error: code = NotFound desc = could not find container \"2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e\": container with ID starting with 2f98a21cfbdc3517c2e65639f180a45021471b405676afd9664e1f227d65ea7e not found: ID does not exist" Nov 03 19:25:08.279506 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.279468 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4"] Nov 03 19:25:08.283658 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:08.283629 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-942af8-78446b5f5f-45fb4"] Nov 03 19:25:09.272436 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:09.272404 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fb572c-57ca-401c-8222-37dfb13aa9f7" path="/var/lib/kubelet/pods/a0fb572c-57ca-401c-8222-37dfb13aa9f7/volumes" Nov 03 19:25:10.330529 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:10.330489 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:10.927640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:10.927606 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:11.605129 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:11.605073 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:11.814971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:11.814935 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:12.020458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:12.020370 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:12.163501 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:12.163464 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk"] Nov 03 19:25:12.209883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:12.209845 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-942af8-qfhgk"] Nov 03 19:25:12.672655 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:12.672624 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:13.271994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:13.271954 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be49e3f-a674-48b4-9d69-cb5db05fe570" path="/var/lib/kubelet/pods/1be49e3f-a674-48b4-9d69-cb5db05fe570/volumes" Nov 03 19:25:16.913887 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:16.913850 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:17.536611 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:17.536581 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:17.753493 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:17.753444 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:17.978052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:17.978015 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:18.226573 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:18.226535 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:25:18.358290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:18.358246 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:19.802246 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:19.802216 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:20.009646 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.009602 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:20.062387 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.062304 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg"] Nov 03 19:25:20.062557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.062514 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" containerID="cri-o://b97b1be2735ded6d25b6a60eee4933f865ef1ab34bc7bf2c23bf70130a66daa4" gracePeriod=30 Nov 03 19:25:20.259194 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.259160 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:20.919496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.919454 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8"] Nov 03 19:25:20.919934 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.919919 5988 cpu_manager.go:414] "containerMap: removing container" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" Nov 03 19:25:20.919985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.919934 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" Nov 03 19:25:20.919985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.919959 5988 cpu_manager.go:414] "containerMap: removing container" podUID="1be49e3f-a674-48b4-9d69-cb5db05fe570" containerName="revoke-keys" Nov 03 19:25:20.919985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.919964 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be49e3f-a674-48b4-9d69-cb5db05fe570" containerName="revoke-keys" Nov 03 19:25:20.919985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.919978 5988 cpu_manager.go:414] "containerMap: removing container" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" Nov 03 19:25:20.919985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.919985 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" Nov 03 19:25:20.920148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.920004 5988 cpu_manager.go:414] "containerMap: removing container" podUID="a0fb572c-57ca-401c-8222-37dfb13aa9f7" containerName="payload-processing" Nov 03 19:25:20.920148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.920010 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fb572c-57ca-401c-8222-37dfb13aa9f7" containerName="payload-processing" Nov 03 19:25:20.920148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.920076 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be49e3f-a674-48b4-9d69-cb5db05fe570" containerName="revoke-keys" Nov 03 19:25:20.920148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.920086 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" Nov 03 19:25:20.920148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.920094 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fb572c-57ca-401c-8222-37dfb13aa9f7" containerName="payload-processing" Nov 03 19:25:20.920148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.920102 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="807178f7-57ea-44bc-94ed-56e4f3ea3cae" containerName="istio-proxy" Nov 03 19:25:20.925971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.925936 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:20.978423 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:20.978369 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8"] Nov 03 19:25:21.000346 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.000310 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.000535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.000359 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6ch\" (UniqueName: \"kubernetes.io/projected/94586e4f-91d1-4ea5-b5ee-ff27007201ec-kube-api-access-2t6ch\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.000535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.000515 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.022646 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.022610 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:25:21.022841 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.022667 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:25:21.101784 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.101746 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.101959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.101825 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.101959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.101851 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6ch\" (UniqueName: \"kubernetes.io/projected/94586e4f-91d1-4ea5-b5ee-ff27007201ec-kube-api-access-2t6ch\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.102445 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.102420 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.104312 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.104281 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.159891 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.159856 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6ch\" (UniqueName: \"kubernetes.io/projected/94586e4f-91d1-4ea5-b5ee-ff27007201ec-kube-api-access-2t6ch\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.241568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.241488 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:21.613656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.613629 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8"] Nov 03 19:25:21.616723 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:25:21.616678 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94586e4f_91d1_4ea5_b5ee_ff27007201ec.slice/crio-cd027efe17877adae7be226b271967ef14218fe55d188bcd073bce483e8d1437 WatchSource:0}: Error finding container cd027efe17877adae7be226b271967ef14218fe55d188bcd073bce483e8d1437: Status 404 returned error can't find the container with id cd027efe17877adae7be226b271967ef14218fe55d188bcd073bce483e8d1437 Nov 03 19:25:21.745060 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:21.745027 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:22.212379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:22.212297 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" event={"ID":"94586e4f-91d1-4ea5-b5ee-ff27007201ec","Type":"ContainerStarted","Data":"16ded19dce8f2aba7017deede7a3ac18dd21c95e499eff80ef3aeb360ff34f7d"} Nov 03 19:25:22.212379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:22.212355 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" event={"ID":"94586e4f-91d1-4ea5-b5ee-ff27007201ec","Type":"ContainerStarted","Data":"cd027efe17877adae7be226b271967ef14218fe55d188bcd073bce483e8d1437"} Nov 03 19:25:22.322580 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:22.322529 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" podStartSLOduration=2.322513306 podStartE2EDuration="2.322513306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:22.321888113 +0000 UTC m=+2797.203304216" watchObservedRunningTime="2026-11-03 19:25:22.322513306 +0000 UTC m=+2797.203929407" Nov 03 19:25:23.220308 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:23.220272 5988 generic.go:356] "Generic (PLEG): container finished" podID="94586e4f-91d1-4ea5-b5ee-ff27007201ec" containerID="16ded19dce8f2aba7017deede7a3ac18dd21c95e499eff80ef3aeb360ff34f7d" exitCode=0 Nov 03 19:25:23.220733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:23.220350 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" event={"ID":"94586e4f-91d1-4ea5-b5ee-ff27007201ec","Type":"ContainerDied","Data":"16ded19dce8f2aba7017deede7a3ac18dd21c95e499eff80ef3aeb360ff34f7d"} Nov 03 19:25:23.249883 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:23.249845 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646"] Nov 03 19:25:23.296404 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:23.296375 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8"] Nov 03 19:25:23.406131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:23.406098 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:24.137270 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.137185 5988 patch_prober.go:29] interesting pod/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:25:24.137270 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.137255 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:25:24.369109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.369079 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:24.538255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.538184 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-xks-ca\") pod \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " Nov 03 19:25:24.538400 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.538261 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-openshift-ca\") pod \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " Nov 03 19:25:24.538400 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.538362 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94586e4f-91d1-4ea5-b5ee-ff27007201ec-kube-api-access-2t6ch\" (UniqueName: \"kubernetes.io/projected/94586e4f-91d1-4ea5-b5ee-ff27007201ec-kube-api-access-2t6ch\") pod \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\" (UID: \"94586e4f-91d1-4ea5-b5ee-ff27007201ec\") " Nov 03 19:25:24.538633 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.538610 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-openshift-ca" pod "94586e4f-91d1-4ea5-b5ee-ff27007201ec" (UID: "94586e4f-91d1-4ea5-b5ee-ff27007201ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:24.540390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.540363 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94586e4f-91d1-4ea5-b5ee-ff27007201ec-kube-api-access-2t6ch" pod "94586e4f-91d1-4ea5-b5ee-ff27007201ec" (UID: "94586e4f-91d1-4ea5-b5ee-ff27007201ec"). InnerVolumeSpecName "kube-api-access-2t6ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:24.540390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.540367 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-xks-ca" pod "94586e4f-91d1-4ea5-b5ee-ff27007201ec" (UID: "94586e4f-91d1-4ea5-b5ee-ff27007201ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:24.639981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.639954 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t6ch\" (UniqueName: \"kubernetes.io/projected/94586e4f-91d1-4ea5-b5ee-ff27007201ec-kube-api-access-2t6ch\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:24.639981 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.639980 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:24.640143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:24.639991 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94586e4f-91d1-4ea5-b5ee-ff27007201ec-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.235820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.235791 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" Nov 03 19:25:25.235969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.235797 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8" event={"ID":"94586e4f-91d1-4ea5-b5ee-ff27007201ec","Type":"ContainerDied","Data":"cd027efe17877adae7be226b271967ef14218fe55d188bcd073bce483e8d1437"} Nov 03 19:25:25.235969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.235921 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd027efe17877adae7be226b271967ef14218fe55d188bcd073bce483e8d1437" Nov 03 19:25:25.238541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.238516 5988 generic.go:356] "Generic (PLEG): container finished" podID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerID="b97b1be2735ded6d25b6a60eee4933f865ef1ab34bc7bf2c23bf70130a66daa4" exitCode=0 Nov 03 19:25:25.238697 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.238586 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" event={"ID":"b67e964c-219b-43c9-9e23-0e1aa854fbff","Type":"ContainerDied","Data":"b97b1be2735ded6d25b6a60eee4933f865ef1ab34bc7bf2c23bf70130a66daa4"} Nov 03 19:25:25.238697 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.238643 5988 scope.go:122] "RemoveContainer" containerID="1efeb9a34e39b9fc08ff33b8092f496221d04043f596a483c21091f06902a992" Nov 03 19:25:25.306692 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.306656 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:25:25.447178 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447110 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67e964c-219b-43c9-9e23-0e1aa854fbff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67e964c-219b-43c9-9e23-0e1aa854fbff-istiod-ca-cert\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447192 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-certs\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447259 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-token\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-token\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447305 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-podinfo\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447334 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-kube-api-access-hpv2q\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-kube-api-access-hpv2q\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447404 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-socket\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447467 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-credential-socket\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447476 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-certs" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447476 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67e964c-219b-43c9-9e23-0e1aa854fbff-istiod-ca-cert" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447509 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-data\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.447605 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447564 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-envoy\") pod \"b67e964c-219b-43c9-9e23-0e1aa854fbff\" (UID: \"b67e964c-219b-43c9-9e23-0e1aa854fbff\") " Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447714 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-socket" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447915 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-data" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.447929 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-credential-socket" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.448059 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.448079 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.448094 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.448108 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67e964c-219b-43c9-9e23-0e1aa854fbff-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.448188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.448120 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.449734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.449710 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-kube-api-access-hpv2q" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "kube-api-access-hpv2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:25.449922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.449899 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-podinfo" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:25:25.450026 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.450010 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-envoy" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:25.450321 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.450304 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-token" pod "b67e964c-219b-43c9-9e23-0e1aa854fbff" (UID: "b67e964c-219b-43c9-9e23-0e1aa854fbff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:25.549147 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.549122 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.549147 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.549146 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpv2q\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-kube-api-access-hpv2q\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.549280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.549157 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.549280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:25.549166 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67e964c-219b-43c9-9e23-0e1aa854fbff-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:25.936497 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:25:25.936464 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:25:26.245601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:26.245528 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" Nov 03 19:25:26.245776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:26.245522 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg" event={"ID":"b67e964c-219b-43c9-9e23-0e1aa854fbff","Type":"ContainerDied","Data":"74d5e5c14450625bf7a40d06e6541f73ef84ecc9cea0a1195c46294f8c32704d"} Nov 03 19:25:26.245776 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:26.245665 5988 scope.go:122] "RemoveContainer" containerID="b97b1be2735ded6d25b6a60eee4933f865ef1ab34bc7bf2c23bf70130a66daa4" Nov 03 19:25:26.445929 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:26.445898 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg"] Nov 03 19:25:26.492298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:26.492268 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-942af8-openshift-default-878544c45-bpzgg"] Nov 03 19:25:26.612998 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:26.612967 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:27.272332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:27.272299 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" path="/var/lib/kubelet/pods/b67e964c-219b-43c9-9e23-0e1aa854fbff/volumes" Nov 03 19:25:30.343788 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:30.343751 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4"] Nov 03 19:25:30.789673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:30.789599 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:30.993186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:30.993150 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:31.207957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:31.207923 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:33.325117 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:33.325086 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:33.580004 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:33.579915 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:35.124437 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.124404 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:35.344708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.344645 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" podUID="fa7aae1f-b048-4e60-94b4-0831a6e24c92" containerName="payload-processing" containerID="cri-o://611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa" gracePeriod=25 Nov 03 19:25:35.592174 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.592145 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 19:25:35.641058 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.641025 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7aae1f-b048-4e60-94b4-0831a6e24c92-kube-api-access-9zxwm\" (UniqueName: \"kubernetes.io/projected/fa7aae1f-b048-4e60-94b4-0831a6e24c92-kube-api-access-9zxwm\") pod \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " Nov 03 19:25:35.641208 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.641185 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7aae1f-b048-4e60-94b4-0831a6e24c92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa7aae1f-b048-4e60-94b4-0831a6e24c92-plugins-config-volume\") pod \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\" (UID: \"fa7aae1f-b048-4e60-94b4-0831a6e24c92\") " Nov 03 19:25:35.641606 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.641582 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7aae1f-b048-4e60-94b4-0831a6e24c92-plugins-config-volume" pod "fa7aae1f-b048-4e60-94b4-0831a6e24c92" (UID: "fa7aae1f-b048-4e60-94b4-0831a6e24c92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:35.643116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.643095 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7aae1f-b048-4e60-94b4-0831a6e24c92-kube-api-access-9zxwm" pod "fa7aae1f-b048-4e60-94b4-0831a6e24c92" (UID: "fa7aae1f-b048-4e60-94b4-0831a6e24c92"). InnerVolumeSpecName "kube-api-access-9zxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:35.742020 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.741953 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa7aae1f-b048-4e60-94b4-0831a6e24c92-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:35.742020 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:35.741979 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxwm\" (UniqueName: \"kubernetes.io/projected/fa7aae1f-b048-4e60-94b4-0831a6e24c92-kube-api-access-9zxwm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:36.302093 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.302059 5988 generic.go:356] "Generic (PLEG): container finished" podID="fa7aae1f-b048-4e60-94b4-0831a6e24c92" containerID="611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa" exitCode=0 Nov 03 19:25:36.302576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.302109 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" Nov 03 19:25:36.302576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.302148 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" event={"ID":"fa7aae1f-b048-4e60-94b4-0831a6e24c92","Type":"ContainerDied","Data":"611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa"} Nov 03 19:25:36.302576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.302213 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4" event={"ID":"fa7aae1f-b048-4e60-94b4-0831a6e24c92","Type":"ContainerDied","Data":"e09ade9e11b1b355082b29673ba8ba36c8592d7a4e542de26b60355ee3d1a3e3"} Nov 03 19:25:36.302576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.302247 5988 scope.go:122] "RemoveContainer" containerID="611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa" Nov 03 19:25:36.312724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.312703 5988 scope.go:122] "RemoveContainer" containerID="611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa" Nov 03 19:25:36.312971 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:25:36.312952 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa\": container with ID starting with 611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa not found: ID does not exist" containerID="611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa" Nov 03 19:25:36.313048 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.312985 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa"} err="failed to get container status \"611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa\": rpc error: code = NotFound desc = could not find container \"611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa\": container with ID starting with 611b30f461992a9be54a9b4c244581bf8fa74cb92b54654b5f01e9b7e213d8aa not found: ID does not exist" Nov 03 19:25:36.456999 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.456963 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4"] Nov 03 19:25:36.503375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.503340 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-355e1a-57bbcfbbb9-ggfc4"] Nov 03 19:25:36.800350 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:36.800312 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:37.271708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:37.271650 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7aae1f-b048-4e60-94b4-0831a6e24c92" path="/var/lib/kubelet/pods/fa7aae1f-b048-4e60-94b4-0831a6e24c92/volumes" Nov 03 19:25:38.270832 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.270796 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" containerID="cri-o://af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3" gracePeriod=45 Nov 03 19:25:38.316075 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.316037 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" containerID="cri-o://4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c" gracePeriod=45 Nov 03 19:25:38.566163 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.566139 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:25:38.587101 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.587071 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:25:38.671666 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.671633 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-dshm\") pod \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " Nov 03 19:25:38.671666 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.671668 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tmp-dir\") pod \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " Nov 03 19:25:38.671874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.671721 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-dshm\") pod \"65d04039-cc62-4e81-b1e6-8d0d58b41704\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " Nov 03 19:25:38.671874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.671754 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d04039-cc62-4e81-b1e6-8d0d58b41704-tls-certs\" (UniqueName: \"kubernetes.io/secret/65d04039-cc62-4e81-b1e6-8d0d58b41704-tls-certs\") pod \"65d04039-cc62-4e81-b1e6-8d0d58b41704\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " Nov 03 19:25:38.671874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.671808 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-model-cache\") pod \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " Nov 03 19:25:38.671874 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.671847 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-tmp-dir\") pod \"65d04039-cc62-4e81-b1e6-8d0d58b41704\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " Nov 03 19:25:38.672023 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.671884 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d04039-cc62-4e81-b1e6-8d0d58b41704-kube-api-access-hvnww\" (UniqueName: \"kubernetes.io/projected/65d04039-cc62-4e81-b1e6-8d0d58b41704-kube-api-access-hvnww\") pod \"65d04039-cc62-4e81-b1e6-8d0d58b41704\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " Nov 03 19:25:38.672082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672054 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tls-certs\") pod \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " Nov 03 19:25:38.672143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672126 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-home\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-home\") pod \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " Nov 03 19:25:38.672201 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672172 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-home\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-home\") pod \"65d04039-cc62-4e81-b1e6-8d0d58b41704\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " Nov 03 19:25:38.672260 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672217 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-model-cache\") pod \"65d04039-cc62-4e81-b1e6-8d0d58b41704\" (UID: \"65d04039-cc62-4e81-b1e6-8d0d58b41704\") " Nov 03 19:25:38.672260 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672255 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-kube-api-access-kbc5k\" (UniqueName: \"kubernetes.io/projected/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-kube-api-access-kbc5k\") pod \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\" (UID: \"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b\") " Nov 03 19:25:38.672375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672280 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-model-cache" pod "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" (UID: "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.672507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672482 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-tmp-dir" pod "65d04039-cc62-4e81-b1e6-8d0d58b41704" (UID: "65d04039-cc62-4e81-b1e6-8d0d58b41704"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.672622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672606 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.672706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672627 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.672775 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672706 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-home" pod "65d04039-cc62-4e81-b1e6-8d0d58b41704" (UID: "65d04039-cc62-4e81-b1e6-8d0d58b41704"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.672894 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.672873 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-model-cache" pod "65d04039-cc62-4e81-b1e6-8d0d58b41704" (UID: "65d04039-cc62-4e81-b1e6-8d0d58b41704"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.673052 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.673020 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tmp-dir" pod "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" (UID: "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.673253 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.673231 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-home" pod "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" (UID: "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.674771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.674741 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-dshm" pod "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" (UID: "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.674867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.674741 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-dshm" pod "65d04039-cc62-4e81-b1e6-8d0d58b41704" (UID: "65d04039-cc62-4e81-b1e6-8d0d58b41704"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:25:38.674867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.674773 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d04039-cc62-4e81-b1e6-8d0d58b41704-tls-certs" pod "65d04039-cc62-4e81-b1e6-8d0d58b41704" (UID: "65d04039-cc62-4e81-b1e6-8d0d58b41704"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:38.674867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.674750 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tls-certs" pod "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" (UID: "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:38.674982 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.674867 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d04039-cc62-4e81-b1e6-8d0d58b41704-kube-api-access-hvnww" pod "65d04039-cc62-4e81-b1e6-8d0d58b41704" (UID: "65d04039-cc62-4e81-b1e6-8d0d58b41704"). InnerVolumeSpecName "kube-api-access-hvnww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:38.675271 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.675250 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-kube-api-access-kbc5k" pod "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" (UID: "ae6b35a7-0100-41e8-ad9c-d4e5efd0120b"). InnerVolumeSpecName "kube-api-access-kbc5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:38.773252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773216 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773242 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773252 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773268 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbc5k\" (UniqueName: \"kubernetes.io/projected/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-kube-api-access-kbc5k\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773280 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773288 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773295 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65d04039-cc62-4e81-b1e6-8d0d58b41704-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773303 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65d04039-cc62-4e81-b1e6-8d0d58b41704-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773311 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvnww\" (UniqueName: \"kubernetes.io/projected/65d04039-cc62-4e81-b1e6-8d0d58b41704-kube-api-access-hvnww\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:38.773508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:38.773320 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:39.320991 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.320961 5988 generic.go:356] "Generic (PLEG): container finished" podID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerID="af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3" exitCode=0 Nov 03 19:25:39.321335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.321035 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" Nov 03 19:25:39.321335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.321041 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" event={"ID":"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b","Type":"ContainerDied","Data":"af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3"} Nov 03 19:25:39.321335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.321084 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646" event={"ID":"ae6b35a7-0100-41e8-ad9c-d4e5efd0120b","Type":"ContainerDied","Data":"2575988b05cde4ccb74b7e0865594be489b00a2ecaf2e503d14841f7d7187f0e"} Nov 03 19:25:39.321335 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.321109 5988 scope.go:122] "RemoveContainer" containerID="af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3" Nov 03 19:25:39.322563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.322482 5988 generic.go:356] "Generic (PLEG): container finished" podID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerID="4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c" exitCode=0 Nov 03 19:25:39.322563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.322529 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" event={"ID":"65d04039-cc62-4e81-b1e6-8d0d58b41704","Type":"ContainerDied","Data":"4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c"} Nov 03 19:25:39.322563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.322556 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" event={"ID":"65d04039-cc62-4e81-b1e6-8d0d58b41704","Type":"ContainerDied","Data":"3eb67c2c02e00bd18ef789c88782d242a59dbc97ae2891cf06f20f383fe9bc85"} Nov 03 19:25:39.322563 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.322558 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8" Nov 03 19:25:39.331665 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.331633 5988 scope.go:122] "RemoveContainer" containerID="af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3" Nov 03 19:25:39.331965 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:25:39.331944 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3\": container with ID starting with af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3 not found: ID does not exist" containerID="af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3" Nov 03 19:25:39.332051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.331975 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3"} err="failed to get container status \"af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3\": rpc error: code = NotFound desc = could not find container \"af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3\": container with ID starting with af61f40685fc6a9ef14058fa103bb8e4a63596c78e0f60882abbb43ea54150d3 not found: ID does not exist" Nov 03 19:25:39.332051 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.331995 5988 scope.go:122] "RemoveContainer" containerID="4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c" Nov 03 19:25:39.339902 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.339885 5988 scope.go:122] "RemoveContainer" containerID="4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c" Nov 03 19:25:39.340160 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:25:39.340135 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c\": container with ID starting with 4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c not found: ID does not exist" containerID="4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c" Nov 03 19:25:39.340218 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.340162 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c"} err="failed to get container status \"4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c\": rpc error: code = NotFound desc = could not find container \"4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c\": container with ID starting with 4026c45078c6a074b17624ed1eb43b51485e36fd3dfa2182fb5d86dcfaf6403c not found: ID does not exist" Nov 03 19:25:39.475971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.475936 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646"] Nov 03 19:25:39.522629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.522597 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/distinct-w1-355e1a-kserve-6699c7796-pr646"] Nov 03 19:25:39.583147 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.583119 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8"] Nov 03 19:25:39.588031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:39.588007 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-355e1a/premium-w1-355e1a-kserve-7c74898bb5-vbwk8"] Nov 03 19:25:40.877922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.877884 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr"] Nov 03 19:25:40.878553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878530 5988 cpu_manager.go:414] "containerMap: removing container" podUID="94586e4f-91d1-4ea5-b5ee-ff27007201ec" containerName="revoke-keys" Nov 03 19:25:40.878643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878555 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94586e4f-91d1-4ea5-b5ee-ff27007201ec" containerName="revoke-keys" Nov 03 19:25:40.878643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878573 5988 cpu_manager.go:414] "containerMap: removing container" podUID="fa7aae1f-b048-4e60-94b4-0831a6e24c92" containerName="payload-processing" Nov 03 19:25:40.878643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878581 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7aae1f-b048-4e60-94b4-0831a6e24c92" containerName="payload-processing" Nov 03 19:25:40.878643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878621 5988 cpu_manager.go:414] "containerMap: removing container" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" Nov 03 19:25:40.878643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878629 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" Nov 03 19:25:40.878643 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878642 5988 cpu_manager.go:414] "containerMap: removing container" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878649 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878720 5988 cpu_manager.go:414] "containerMap: removing container" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878730 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878893 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878907 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" containerName="main" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878922 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="94586e4f-91d1-4ea5-b5ee-ff27007201ec" containerName="revoke-keys" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878933 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7aae1f-b048-4e60-94b4-0831a6e24c92" containerName="payload-processing" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878944 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" containerName="main" Nov 03 19:25:40.879000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.878951 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" Nov 03 19:25:40.883394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.883371 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:40.936975 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.936944 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr"] Nov 03 19:25:40.988794 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.988766 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:25:40.988959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.988857 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:25:40.995609 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.995582 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:40.995783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.995630 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:40.995783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:40.995772 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44n72\" (UniqueName: \"kubernetes.io/projected/88eead85-009a-4b4a-9f64-f22166adcccc-kube-api-access-44n72\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.096864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.096823 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44n72\" (UniqueName: \"kubernetes.io/projected/88eead85-009a-4b4a-9f64-f22166adcccc-kube-api-access-44n72\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.097063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.096943 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.097063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.096974 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.097712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.097668 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.099215 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.099196 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.155375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.155303 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44n72\" (UniqueName: \"kubernetes.io/projected/88eead85-009a-4b4a-9f64-f22166adcccc-kube-api-access-44n72\") pod \"maas-api-revoke-sub-bd8fec9d-d8xnr\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.197551 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.197522 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:41.273787 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.273755 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d04039-cc62-4e81-b1e6-8d0d58b41704" path="/var/lib/kubelet/pods/65d04039-cc62-4e81-b1e6-8d0d58b41704/volumes" Nov 03 19:25:41.274267 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.274249 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6b35a7-0100-41e8-ad9c-d4e5efd0120b" path="/var/lib/kubelet/pods/ae6b35a7-0100-41e8-ad9c-d4e5efd0120b/volumes" Nov 03 19:25:41.564820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.564797 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr"] Nov 03 19:25:41.568015 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:25:41.567988 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eead85_009a_4b4a_9f64_f22166adcccc.slice/crio-668f45534aed76c8d48f395de3427a15252266dcb5a537e1499eed37ec775e07 WatchSource:0}: Error finding container 668f45534aed76c8d48f395de3427a15252266dcb5a537e1499eed37ec775e07: Status 404 returned error can't find the container with id 668f45534aed76c8d48f395de3427a15252266dcb5a537e1499eed37ec775e07 Nov 03 19:25:41.718304 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:41.718265 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:42.074118 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:42.074087 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:25:42.346225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:42.346186 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" event={"ID":"88eead85-009a-4b4a-9f64-f22166adcccc","Type":"ContainerStarted","Data":"16a39ce472063d2b891382a355b7095bbdf7658d3f4d256b01e5faaebe99530a"} Nov 03 19:25:42.346387 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:42.346250 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" event={"ID":"88eead85-009a-4b4a-9f64-f22166adcccc","Type":"ContainerStarted","Data":"668f45534aed76c8d48f395de3427a15252266dcb5a537e1499eed37ec775e07"} Nov 03 19:25:42.456467 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:42.456418 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" podStartSLOduration=2.456390567 podStartE2EDuration="2.456390567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:42.455913657 +0000 UTC m=+2817.337329760" watchObservedRunningTime="2026-11-03 19:25:42.456390567 +0000 UTC m=+2817.337806673" Nov 03 19:25:43.353279 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:43.353244 5988 generic.go:356] "Generic (PLEG): container finished" podID="88eead85-009a-4b4a-9f64-f22166adcccc" containerID="16a39ce472063d2b891382a355b7095bbdf7658d3f4d256b01e5faaebe99530a" exitCode=0 Nov 03 19:25:43.353734 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:43.353327 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" event={"ID":"88eead85-009a-4b4a-9f64-f22166adcccc","Type":"ContainerDied","Data":"16a39ce472063d2b891382a355b7095bbdf7658d3f4d256b01e5faaebe99530a"} Nov 03 19:25:44.483483 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.483459 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:44.533297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.533269 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-openshift-ca\") pod \"88eead85-009a-4b4a-9f64-f22166adcccc\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " Nov 03 19:25:44.533431 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.533400 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88eead85-009a-4b4a-9f64-f22166adcccc-kube-api-access-44n72\" (UniqueName: \"kubernetes.io/projected/88eead85-009a-4b4a-9f64-f22166adcccc-kube-api-access-44n72\") pod \"88eead85-009a-4b4a-9f64-f22166adcccc\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " Nov 03 19:25:44.533489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.533440 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-xks-ca\") pod \"88eead85-009a-4b4a-9f64-f22166adcccc\" (UID: \"88eead85-009a-4b4a-9f64-f22166adcccc\") " Nov 03 19:25:44.533589 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.533566 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-openshift-ca" pod "88eead85-009a-4b4a-9f64-f22166adcccc" (UID: "88eead85-009a-4b4a-9f64-f22166adcccc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:25:44.533721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.533706 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:44.535542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.535521 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-xks-ca" pod "88eead85-009a-4b4a-9f64-f22166adcccc" (UID: "88eead85-009a-4b4a-9f64-f22166adcccc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:25:44.535661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.535547 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eead85-009a-4b4a-9f64-f22166adcccc-kube-api-access-44n72" pod "88eead85-009a-4b4a-9f64-f22166adcccc" (UID: "88eead85-009a-4b4a-9f64-f22166adcccc"). InnerVolumeSpecName "kube-api-access-44n72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:25:44.635290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.635216 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44n72\" (UniqueName: \"kubernetes.io/projected/88eead85-009a-4b4a-9f64-f22166adcccc-kube-api-access-44n72\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:44.635290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:44.635248 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eead85-009a-4b4a-9f64-f22166adcccc-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:25:45.365357 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:45.365327 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" Nov 03 19:25:45.365524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:45.365356 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr" event={"ID":"88eead85-009a-4b4a-9f64-f22166adcccc","Type":"ContainerDied","Data":"668f45534aed76c8d48f395de3427a15252266dcb5a537e1499eed37ec775e07"} Nov 03 19:25:45.365524 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:45.365406 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="668f45534aed76c8d48f395de3427a15252266dcb5a537e1499eed37ec775e07" Nov 03 19:25:46.313451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:46.313416 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:46.494183 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:46.494153 5988 scope.go:122] "RemoveContainer" containerID="47b1dfb14909a796012f317015e7046c97a094716cc39806d3c3c1a112686e78" Nov 03 19:25:46.505439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:46.505416 5988 scope.go:122] "RemoveContainer" containerID="81cea0f17da959333b3989ee769ed09e0c101af8005a15ab7033800ec9ca9d3d" Nov 03 19:25:46.513766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:46.513737 5988 scope.go:122] "RemoveContainer" containerID="a32e185513b11a3f41a35aff852e448050c121100295ac5a2a678e411b883332" Nov 03 19:25:46.521479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:46.521464 5988 scope.go:122] "RemoveContainer" containerID="cd35e4e8c04a21176cb14b0203e61f8decbea833e022f8dd6cba8d25236baa71" Nov 03 19:25:51.891613 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:51.891576 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:52.211442 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:52.211354 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:52.348083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:52.348035 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8"] Nov 03 19:25:52.354946 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:52.354915 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-770e67-j8kw8"] Nov 03 19:25:52.418222 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:52.418172 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:52.865082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:52.865036 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:53.272482 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:53.272407 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94586e4f-91d1-4ea5-b5ee-ff27007201ec" path="/var/lib/kubelet/pods/94586e4f-91d1-4ea5-b5ee-ff27007201ec/volumes" Nov 03 19:25:55.935997 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:25:55.935959 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:25:57.005601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:57.005571 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:57.309950 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:57.309860 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:58.872233 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:58.872205 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:59.084943 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:59.084908 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:25:59.291764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:25:59.291655 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:00.813513 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:00.813475 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:01.084853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:01.084806 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:01.348071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:01.347978 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:01.556115 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:01.556077 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:01.760349 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:01.760259 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:02.640985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:02.640949 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:02.905896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:02.905806 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:03.106316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:03.106281 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:03.213179 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:03.213102 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:26:03.583204 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:03.583163 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:07.260878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:07.260842 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:07.507298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:07.507257 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:08.008653 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:08.008616 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:09.901869 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:09.901830 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:10.111268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:10.111221 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:10.313774 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:10.313665 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:12.067957 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:12.067925 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:12.284873 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:12.284835 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:12.613056 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:12.613021 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:16.904667 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:16.904629 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:17.227234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:17.227147 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:21.227740 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:21.227678 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:22.703880 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:22.703843 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:22.920619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:22.920577 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:23.130297 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:23.130249 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:23.615628 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:23.615589 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:25.936482 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:26:25.936441 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:26:27.716313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:27.716272 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:27.977958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:27.977875 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:26:30.006470 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:30.006432 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:30.212279 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:30.212244 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:30.410636 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:30.410591 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:30.843121 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:30.843011 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:31.050458 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:31.050421 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm"] Nov 03 19:26:31.096621 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:31.096541 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7688d859-g2rkm"] Nov 03 19:26:31.271652 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:31.271615 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fe2b11-c762-4290-a841-e450d67b71cc" path="/var/lib/kubelet/pods/e4fe2b11-c762-4290-a841-e450d67b71cc/volumes" Nov 03 19:26:32.408500 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:32.408471 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:35.754011 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:35.753975 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx"] Nov 03 19:26:35.980414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:35.980372 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:37.303609 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:37.303575 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:41.906914 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:41.906872 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:42.846551 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:42.846512 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv"] Nov 03 19:26:43.284725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:43.284618 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:43.479545 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:43.479505 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:46.404911 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:46.404871 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:26:46.579641 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:46.579605 5988 scope.go:122] "RemoveContainer" containerID="9c0841f6213c48e951522733763ea5f17c0a8ddde4cc054a09eb362ef79bb39c" Nov 03 19:26:46.588804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:46.588780 5988 scope.go:122] "RemoveContainer" containerID="900aec686587c82127611b55d2b3c5f2813b2419748bb0e3eeb0327b59751fdd" Nov 03 19:26:47.847650 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:47.847605 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" podUID="3cf096dc-a935-4576-8fbf-74326af2d009" containerName="payload-processing" containerID="cri-o://5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac" gracePeriod=25 Nov 03 19:26:48.104527 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.104455 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 19:26:48.255148 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.255108 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf096dc-a935-4576-8fbf-74326af2d009-kube-api-access-j87jr\" (UniqueName: \"kubernetes.io/projected/3cf096dc-a935-4576-8fbf-74326af2d009-kube-api-access-j87jr\") pod \"3cf096dc-a935-4576-8fbf-74326af2d009\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " Nov 03 19:26:48.255381 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.255220 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf096dc-a935-4576-8fbf-74326af2d009-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf096dc-a935-4576-8fbf-74326af2d009-plugins-config-volume\") pod \"3cf096dc-a935-4576-8fbf-74326af2d009\" (UID: \"3cf096dc-a935-4576-8fbf-74326af2d009\") " Nov 03 19:26:48.255658 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.255629 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf096dc-a935-4576-8fbf-74326af2d009-plugins-config-volume" pod "3cf096dc-a935-4576-8fbf-74326af2d009" (UID: "3cf096dc-a935-4576-8fbf-74326af2d009"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:26:48.257306 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.257275 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf096dc-a935-4576-8fbf-74326af2d009-kube-api-access-j87jr" pod "3cf096dc-a935-4576-8fbf-74326af2d009" (UID: "3cf096dc-a935-4576-8fbf-74326af2d009"). InnerVolumeSpecName "kube-api-access-j87jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:48.356486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.356382 5988 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cf096dc-a935-4576-8fbf-74326af2d009-plugins-config-volume\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:48.356486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.356420 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j87jr\" (UniqueName: \"kubernetes.io/projected/3cf096dc-a935-4576-8fbf-74326af2d009-kube-api-access-j87jr\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:48.702247 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.702156 5988 generic.go:356] "Generic (PLEG): container finished" podID="3cf096dc-a935-4576-8fbf-74326af2d009" containerID="5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac" exitCode=0 Nov 03 19:26:48.702247 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.702218 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" Nov 03 19:26:48.702471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.702246 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" event={"ID":"3cf096dc-a935-4576-8fbf-74326af2d009","Type":"ContainerDied","Data":"5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac"} Nov 03 19:26:48.702471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.702297 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv" event={"ID":"3cf096dc-a935-4576-8fbf-74326af2d009","Type":"ContainerDied","Data":"1a14dd85650e7d992a6b863e49120015a206ab6ac20b157fb3ae5e20a10c8d6a"} Nov 03 19:26:48.702471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.702319 5988 scope.go:122] "RemoveContainer" containerID="5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac" Nov 03 19:26:48.711865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.711845 5988 scope.go:122] "RemoveContainer" containerID="5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac" Nov 03 19:26:48.712151 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:26:48.712129 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac\": container with ID starting with 5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac not found: ID does not exist" containerID="5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac" Nov 03 19:26:48.712198 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.712162 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac"} err="failed to get container status \"5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac\": rpc error: code = NotFound desc = could not find container \"5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac\": container with ID starting with 5121a375af3673464775f41454980dbd75f74b5335034988c921c7fd537c40ac not found: ID does not exist" Nov 03 19:26:48.856457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.856411 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv"] Nov 03 19:26:48.902808 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:48.902765 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-32a59b-6bd5df9875-tlgmv"] Nov 03 19:26:49.272502 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:49.272458 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf096dc-a935-4576-8fbf-74326af2d009" path="/var/lib/kubelet/pods/3cf096dc-a935-4576-8fbf-74326af2d009/volumes" Nov 03 19:26:50.773948 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:50.773902 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" containerID="cri-o://5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0" gracePeriod=45 Nov 03 19:26:51.019552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.019524 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 19:26:51.185113 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185079 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-tmp-dir\") pod \"c2667b76-561f-4f80-8338-04fb19a5a99a\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " Nov 03 19:26:51.185342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185139 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-dshm\") pod \"c2667b76-561f-4f80-8338-04fb19a5a99a\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " Nov 03 19:26:51.185342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185178 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2667b76-561f-4f80-8338-04fb19a5a99a-kube-api-access-d84cq\" (UniqueName: \"kubernetes.io/projected/c2667b76-561f-4f80-8338-04fb19a5a99a-kube-api-access-d84cq\") pod \"c2667b76-561f-4f80-8338-04fb19a5a99a\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " Nov 03 19:26:51.185342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185225 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-model-cache\") pod \"c2667b76-561f-4f80-8338-04fb19a5a99a\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " Nov 03 19:26:51.185342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185283 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-home\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-home\") pod \"c2667b76-561f-4f80-8338-04fb19a5a99a\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " Nov 03 19:26:51.185618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185352 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2667b76-561f-4f80-8338-04fb19a5a99a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2667b76-561f-4f80-8338-04fb19a5a99a-tls-certs\") pod \"c2667b76-561f-4f80-8338-04fb19a5a99a\" (UID: \"c2667b76-561f-4f80-8338-04fb19a5a99a\") " Nov 03 19:26:51.185618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185396 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-tmp-dir" pod "c2667b76-561f-4f80-8338-04fb19a5a99a" (UID: "c2667b76-561f-4f80-8338-04fb19a5a99a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.185618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185494 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-model-cache" pod "c2667b76-561f-4f80-8338-04fb19a5a99a" (UID: "c2667b76-561f-4f80-8338-04fb19a5a99a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.185618 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185604 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-home" pod "c2667b76-561f-4f80-8338-04fb19a5a99a" (UID: "c2667b76-561f-4f80-8338-04fb19a5a99a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.185857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185790 5988 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-home\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.185857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185835 5988 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-tmp-dir\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.185857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.185852 5988 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-model-cache\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.187699 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.187663 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2667b76-561f-4f80-8338-04fb19a5a99a-tls-certs" pod "c2667b76-561f-4f80-8338-04fb19a5a99a" (UID: "c2667b76-561f-4f80-8338-04fb19a5a99a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:26:51.187836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.187808 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2667b76-561f-4f80-8338-04fb19a5a99a-kube-api-access-d84cq" pod "c2667b76-561f-4f80-8338-04fb19a5a99a" (UID: "c2667b76-561f-4f80-8338-04fb19a5a99a"). InnerVolumeSpecName "kube-api-access-d84cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:26:51.187836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.187814 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-dshm" pod "c2667b76-561f-4f80-8338-04fb19a5a99a" (UID: "c2667b76-561f-4f80-8338-04fb19a5a99a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:26:51.287016 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.286984 5988 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2667b76-561f-4f80-8338-04fb19a5a99a-dshm\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.287016 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.287017 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84cq\" (UniqueName: \"kubernetes.io/projected/c2667b76-561f-4f80-8338-04fb19a5a99a-kube-api-access-d84cq\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.287200 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.287031 5988 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2667b76-561f-4f80-8338-04fb19a5a99a-tls-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:26:51.721577 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.721538 5988 generic.go:356] "Generic (PLEG): container finished" podID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerID="5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0" exitCode=0 Nov 03 19:26:51.721825 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.721591 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" Nov 03 19:26:51.721825 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.721624 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" event={"ID":"c2667b76-561f-4f80-8338-04fb19a5a99a","Type":"ContainerDied","Data":"5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0"} Nov 03 19:26:51.721825 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.721672 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx" event={"ID":"c2667b76-561f-4f80-8338-04fb19a5a99a","Type":"ContainerDied","Data":"e755c1609fb58cafa00c7b2f606b4ecdf48a69e7b7deeb1aa0fee652abf04ea8"} Nov 03 19:26:51.721825 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.721722 5988 scope.go:122] "RemoveContainer" containerID="5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0" Nov 03 19:26:51.730676 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.730652 5988 scope.go:122] "RemoveContainer" containerID="5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0" Nov 03 19:26:51.730989 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:26:51.730969 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0\": container with ID starting with 5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0 not found: ID does not exist" containerID="5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0" Nov 03 19:26:51.731064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.731004 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0"} err="failed to get container status \"5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0\": rpc error: code = NotFound desc = could not find container \"5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0\": container with ID starting with 5a3fe42d6989c1915fdf71df8214165052aed88669cf18a297469d5474464db0 not found: ID does not exist" Nov 03 19:26:51.873878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.873839 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx"] Nov 03 19:26:51.920341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:51.920305 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-32a59b/distinct-w2-32a59b-kserve-77569f9cf9-t48sx"] Nov 03 19:26:53.279733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:53.279694 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" path="/var/lib/kubelet/pods/c2667b76-561f-4f80-8338-04fb19a5a99a/volumes" Nov 03 19:26:55.936588 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:26:55.936504 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:26:57.682221 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:26:57.682183 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:04.044864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:04.044819 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7"] Nov 03 19:27:04.091659 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:04.091626 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd72aefc-td5j7"] Nov 03 19:27:05.272979 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:05.272942 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c8cf53-a227-422a-91c1-61193a15b2e9" path="/var/lib/kubelet/pods/f4c8cf53-a227-422a-91c1-61193a15b2e9/volumes" Nov 03 19:27:05.679624 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:05.679591 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k"] Nov 03 19:27:05.679839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:05.679799 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" podUID="f118efc3-0ece-4311-a4be-6432b720e19a" containerName="maas-api" containerID="cri-o://3895ef973bf0480c8436004240761e2fd68ebec8dcbaae9ec9b098220f98a05c" gracePeriod=30 Nov 03 19:27:05.798766 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:05.798729 5988 generic.go:356] "Generic (PLEG): container finished" podID="f118efc3-0ece-4311-a4be-6432b720e19a" containerID="3895ef973bf0480c8436004240761e2fd68ebec8dcbaae9ec9b098220f98a05c" exitCode=0 Nov 03 19:27:05.798877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:05.798799 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" event={"ID":"f118efc3-0ece-4311-a4be-6432b720e19a","Type":"ContainerDied","Data":"3895ef973bf0480c8436004240761e2fd68ebec8dcbaae9ec9b098220f98a05c"} Nov 03 19:27:05.902280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:05.902247 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:05.938353 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:05.938279 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 19:27:06.034383 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.034345 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f118efc3-0ece-4311-a4be-6432b720e19a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f118efc3-0ece-4311-a4be-6432b720e19a-openshift-service-ca\") pod \"f118efc3-0ece-4311-a4be-6432b720e19a\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " Nov 03 19:27:06.034568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.034408 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-metrics-certs\") pod \"f118efc3-0ece-4311-a4be-6432b720e19a\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " Nov 03 19:27:06.034568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.034476 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f118efc3-0ece-4311-a4be-6432b720e19a-kube-api-access-wgr25\" (UniqueName: \"kubernetes.io/projected/f118efc3-0ece-4311-a4be-6432b720e19a-kube-api-access-wgr25\") pod \"f118efc3-0ece-4311-a4be-6432b720e19a\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " Nov 03 19:27:06.034665 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.034568 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls\") pod \"f118efc3-0ece-4311-a4be-6432b720e19a\" (UID: \"f118efc3-0ece-4311-a4be-6432b720e19a\") " Nov 03 19:27:06.034861 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.034831 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f118efc3-0ece-4311-a4be-6432b720e19a-openshift-service-ca" pod "f118efc3-0ece-4311-a4be-6432b720e19a" (UID: "f118efc3-0ece-4311-a4be-6432b720e19a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:06.036661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.036627 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f118efc3-0ece-4311-a4be-6432b720e19a-kube-api-access-wgr25" pod "f118efc3-0ece-4311-a4be-6432b720e19a" (UID: "f118efc3-0ece-4311-a4be-6432b720e19a"). InnerVolumeSpecName "kube-api-access-wgr25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:06.036817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.036797 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-metrics-certs" pod "f118efc3-0ece-4311-a4be-6432b720e19a" (UID: "f118efc3-0ece-4311-a4be-6432b720e19a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:06.036923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.036905 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls" pod "f118efc3-0ece-4311-a4be-6432b720e19a" (UID: "f118efc3-0ece-4311-a4be-6432b720e19a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:27:06.136188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.136144 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-maas-api-tls\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:06.136188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.136182 5988 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f118efc3-0ece-4311-a4be-6432b720e19a-openshift-service-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:06.136188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.136192 5988 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f118efc3-0ece-4311-a4be-6432b720e19a-metrics-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:06.136446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.136200 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgr25\" (UniqueName: \"kubernetes.io/projected/f118efc3-0ece-4311-a4be-6432b720e19a-kube-api-access-wgr25\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:06.808081 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.808039 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" Nov 03 19:27:06.808081 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.808054 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k" event={"ID":"f118efc3-0ece-4311-a4be-6432b720e19a","Type":"ContainerDied","Data":"1ab05c82a518d1a4e4233729483469bafee240dcf18938e2a7df644563580e47"} Nov 03 19:27:06.808668 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.808140 5988 scope.go:122] "RemoveContainer" containerID="3895ef973bf0480c8436004240761e2fd68ebec8dcbaae9ec9b098220f98a05c" Nov 03 19:27:06.963875 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:06.963831 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k"] Nov 03 19:27:07.011234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:07.011196 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-32a59b-7869497b87-fl25k"] Nov 03 19:27:07.271277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:07.271235 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f118efc3-0ece-4311-a4be-6432b720e19a" path="/var/lib/kubelet/pods/f118efc3-0ece-4311-a4be-6432b720e19a/volumes" Nov 03 19:27:07.318940 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:07.318896 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:27:07.351844 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:07.351813 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:07.580969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:07.580931 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:08.224060 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:08.224019 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:13.216342 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:13.216306 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:13.456733 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:13.456678 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:13.793188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:13.793149 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:14.796773 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:14.796735 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:15.022480 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:15.022443 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:15.094067 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:15.094035 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9"] Nov 03 19:27:15.094310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:15.094274 5988 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" containerID="cri-o://73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630" gracePeriod=30 Nov 03 19:27:15.235969 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:15.235932 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:19.583185 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:19.583150 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:20.350428 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.350397 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:27:20.480134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480032 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istiod-ca-cert\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480134 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480096 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-token\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-token\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480163 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-envoy\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480201 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-data\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480222 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-certs\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480270 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-socket\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480306 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-credential-socket\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480363 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-podinfo\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480408 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480393 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-kube-api-access-6xj6p\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-kube-api-access-6xj6p\") pod \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\" (UID: \"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd\") " Nov 03 19:27:20.480815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480464 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istiod-ca-cert" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:27:20.480815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480587 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-data" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:20.480815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480559 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-certs" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:20.480815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480576 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-socket" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:20.480815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480768 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-credential-socket" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:20.480815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480777 5988 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-data\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.481083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480830 5988 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-certs\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.481083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480851 5988 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-workload-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.481083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.480867 5988 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istiod-ca-cert\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.482673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.482646 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-kube-api-access-6xj6p" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "kube-api-access-6xj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:20.482673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.482658 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-podinfo" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:27:20.482917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.482900 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-envoy" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:27:20.482993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.482900 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-token" pod "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" (UID: "f11ce5f1-b784-41dd-a73d-3e5f5d196dbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:27:20.581804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.581761 5988 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-envoy\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.581804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.581801 5988 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-credential-socket\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.581804 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.581811 5988 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-podinfo\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.582082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.581819 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xj6p\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-kube-api-access-6xj6p\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.582082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.581830 5988 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd-istio-token\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:27:20.892910 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.892863 5988 generic.go:356] "Generic (PLEG): container finished" podID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerID="73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630" exitCode=0 Nov 03 19:27:20.893367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.892917 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" Nov 03 19:27:20.893367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.892946 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" event={"ID":"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd","Type":"ContainerDied","Data":"73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630"} Nov 03 19:27:20.893367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.893006 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9" event={"ID":"f11ce5f1-b784-41dd-a73d-3e5f5d196dbd","Type":"ContainerDied","Data":"d2fdc76e2504a94999e5a60a3389bf24c483d501b375b48292987d45e4631412"} Nov 03 19:27:20.893367 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.893030 5988 scope.go:122] "RemoveContainer" containerID="73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630" Nov 03 19:27:20.902811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.902789 5988 scope.go:122] "RemoveContainer" containerID="73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630" Nov 03 19:27:20.903094 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:27:20.903074 5988 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630\": container with ID starting with 73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630 not found: ID does not exist" containerID="73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630" Nov 03 19:27:20.903138 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:20.903107 5988 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630"} err="failed to get container status \"73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630\": rpc error: code = NotFound desc = could not find container \"73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630\": container with ID starting with 73f30147a4077581eb4d9f5a733f88f03113bce1c567b13d4f6213de658a0630 not found: ID does not exist" Nov 03 19:27:21.051129 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:21.051081 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9"] Nov 03 19:27:21.093932 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:21.093893 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-32a59b-openshift-default-86974868b5-n6kr9"] Nov 03 19:27:21.271364 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:21.271279 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" path="/var/lib/kubelet/pods/f11ce5f1-b784-41dd-a73d-3e5f5d196dbd/volumes" Nov 03 19:27:22.609219 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:22.609181 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:22.808156 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:22.808125 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:22.897247 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:22.897161 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:27:25.935508 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:27:25.935470 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:27:39.425744 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.425703 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx"] Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426127 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426138 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426163 5988 cpu_manager.go:414] "containerMap: removing container" podUID="88eead85-009a-4b4a-9f64-f22166adcccc" containerName="revoke-keys" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426168 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88eead85-009a-4b4a-9f64-f22166adcccc" containerName="revoke-keys" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426184 5988 cpu_manager.go:414] "containerMap: removing container" podUID="f118efc3-0ece-4311-a4be-6432b720e19a" containerName="maas-api" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426191 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f118efc3-0ece-4311-a4be-6432b720e19a" containerName="maas-api" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426206 5988 cpu_manager.go:414] "containerMap: removing container" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426211 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67e964c-219b-43c9-9e23-0e1aa854fbff" containerName="istio-proxy" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426218 5988 cpu_manager.go:414] "containerMap: removing container" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426223 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426235 5988 cpu_manager.go:414] "containerMap: removing container" podUID="3cf096dc-a935-4576-8fbf-74326af2d009" containerName="payload-processing" Nov 03 19:27:39.426255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426240 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf096dc-a935-4576-8fbf-74326af2d009" containerName="payload-processing" Nov 03 19:27:39.426669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426314 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2667b76-561f-4f80-8338-04fb19a5a99a" containerName="main" Nov 03 19:27:39.426669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426326 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf096dc-a935-4576-8fbf-74326af2d009" containerName="payload-processing" Nov 03 19:27:39.426669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426334 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11ce5f1-b784-41dd-a73d-3e5f5d196dbd" containerName="istio-proxy" Nov 03 19:27:39.426669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426344 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="88eead85-009a-4b4a-9f64-f22166adcccc" containerName="revoke-keys" Nov 03 19:27:39.426669 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.426356 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="f118efc3-0ece-4311-a4be-6432b720e19a" containerName="maas-api" Nov 03 19:27:39.431376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.431346 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455414 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455451 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455454 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71191280-54d7-4559-81da-1996a3718675-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455485 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71191280-54d7-4559-81da-1996a3718675-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455642 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455729 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455674 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455749 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhjd\" (UniqueName: \"kubernetes.io/projected/71191280-54d7-4559-81da-1996a3718675-kube-api-access-mvhjd\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455819 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71191280-54d7-4559-81da-1996a3718675-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455840 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.455985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.455861 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.472219 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.472120 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d"] Nov 03 19:27:39.475897 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.475880 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.488565 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.488538 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h6x4t\"" Nov 03 19:27:39.488726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.488538 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 19:27:39.528756 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.528714 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx"] Nov 03 19:27:39.557275 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557216 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.557478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557384 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.557554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557492 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71191280-54d7-4559-81da-1996a3718675-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.557554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557533 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.557709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557562 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.557709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557583 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.557709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557606 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71191280-54d7-4559-81da-1996a3718675-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.557709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557626 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.557709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557646 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.557709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557669 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557751 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71191280-54d7-4559-81da-1996a3718675-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557801 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557854 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557883 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxgm\" (UniqueName: \"kubernetes.io/projected/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-kube-api-access-cjxgm\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557916 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557945 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.557977 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558057 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.558012 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhjd\" (UniqueName: \"kubernetes.io/projected/71191280-54d7-4559-81da-1996a3718675-kube-api-access-mvhjd\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.558104 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.558143 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558448 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.558396 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558638 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.558524 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.558974 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.558957 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71191280-54d7-4559-81da-1996a3718675-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.560108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.560090 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71191280-54d7-4559-81da-1996a3718675-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.560532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.560504 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71191280-54d7-4559-81da-1996a3718675-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.578135 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.578098 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d"] Nov 03 19:27:39.615912 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.615871 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71191280-54d7-4559-81da-1996a3718675-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.616083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.615948 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhjd\" (UniqueName: \"kubernetes.io/projected/71191280-54d7-4559-81da-1996a3718675-kube-api-access-mvhjd\") pod \"maas-default-gateway-openshift-default-7854949c9-hvlvx\" (UID: \"71191280-54d7-4559-81da-1996a3718675\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.659622 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659581 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.659824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659665 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.659824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659714 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxgm\" (UniqueName: \"kubernetes.io/projected/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-kube-api-access-cjxgm\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.659824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659747 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.659824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659814 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660059 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659843 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660059 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659918 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660184 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.660140 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660249 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.660210 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660594 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.660561 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.660623 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.659944 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660762 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.660667 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.660978 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.660960 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.662943 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.662466 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.662943 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.662887 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.718122 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.718049 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.718260 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.718190 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxgm\" (UniqueName: \"kubernetes.io/projected/54551a8b-0cf3-4824-b9ab-6e8858ec5afe-kube-api-access-cjxgm\") pod \"maas-default-gateway-openshift-default-7854949c9-q7v6d\" (UID: \"54551a8b-0cf3-4824-b9ab-6e8858ec5afe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:39.745048 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.745011 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:39.788839 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:39.788806 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:40.168141 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.168096 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx"] Nov 03 19:27:40.170726 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:27:40.170670 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71191280_54d7_4559_81da_1996a3718675.slice/crio-70a1c4f6ecd39b14dd9472f0b5ad6f83ca60399f8f826086d30ed4e5c8607797 WatchSource:0}: Error finding container 70a1c4f6ecd39b14dd9472f0b5ad6f83ca60399f8f826086d30ed4e5c8607797: Status 404 returned error can't find the container with id 70a1c4f6ecd39b14dd9472f0b5ad6f83ca60399f8f826086d30ed4e5c8607797 Nov 03 19:27:40.173100 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.173064 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:27:40.173196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.173130 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:27:40.173196 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.173159 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:27:40.215211 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.215178 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d"] Nov 03 19:27:40.218286 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:27:40.218256 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54551a8b_0cf3_4824_b9ab_6e8858ec5afe.slice/crio-5093664c380f8cb3624e4bea2c8256a446ba2816dbc70e8978dd560c3b83fb83 WatchSource:0}: Error finding container 5093664c380f8cb3624e4bea2c8256a446ba2816dbc70e8978dd560c3b83fb83: Status 404 returned error can't find the container with id 5093664c380f8cb3624e4bea2c8256a446ba2816dbc70e8978dd560c3b83fb83 Nov 03 19:27:40.220936 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.220905 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:27:40.221017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.220970 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:27:40.221017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:40.221005 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:27:41.003727 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.003335 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" event={"ID":"71191280-54d7-4559-81da-1996a3718675","Type":"ContainerStarted","Data":"9d54c83c55341b439b9effd56a42afbaef57b8b06f4384023e3360d4eb657c46"} Nov 03 19:27:41.003727 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.003404 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" event={"ID":"71191280-54d7-4559-81da-1996a3718675","Type":"ContainerStarted","Data":"70a1c4f6ecd39b14dd9472f0b5ad6f83ca60399f8f826086d30ed4e5c8607797"} Nov 03 19:27:41.005143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.005116 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" event={"ID":"54551a8b-0cf3-4824-b9ab-6e8858ec5afe","Type":"ContainerStarted","Data":"d81a2fff4e56101e40c2b8874421561897dbac75e53e0c7295ebab8cddea175d"} Nov 03 19:27:41.005261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.005157 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" event={"ID":"54551a8b-0cf3-4824-b9ab-6e8858ec5afe","Type":"ContainerStarted","Data":"5093664c380f8cb3624e4bea2c8256a446ba2816dbc70e8978dd560c3b83fb83"} Nov 03 19:27:41.098271 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.098228 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:41.120582 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.120524 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podStartSLOduration=2.120511437 podStartE2EDuration="2.120511437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:41.119975025 +0000 UTC m=+2936.001391127" watchObservedRunningTime="2026-11-03 19:27:41.120511437 +0000 UTC m=+2936.001927538" Nov 03 19:27:41.238326 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.238261 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podStartSLOduration=2.238243497 podStartE2EDuration="2.238243497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:41.237495857 +0000 UTC m=+2936.118911956" watchObservedRunningTime="2026-11-03 19:27:41.238243497 +0000 UTC m=+2936.119659601" Nov 03 19:27:41.753709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.747622 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:41.754180 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.754146 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:27:41.754371 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.754345 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 19:27:41.791374 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.791339 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:41.791752 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.791731 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 19:27:41.791922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:41.791897 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 19:27:42.749709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:42.745865 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:27:42.749709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:42.745949 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 19:27:42.795710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:42.790131 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 19:27:42.795710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:42.790207 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 19:27:43.750186 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:43.750143 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:27:43.750799 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:43.750765 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 19:27:43.792710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:43.792152 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 19:27:43.792710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:43.792219 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 19:27:44.756614 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:44.756142 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:27:44.756614 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:44.756212 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 19:27:44.793315 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:44.793175 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 19:27:44.793315 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:44.793240 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 19:27:45.745742 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:45.745662 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:27:45.746067 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:45.746033 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 19:27:45.790178 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:45.790135 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 19:27:45.790857 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:45.790814 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 19:27:46.614528 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:46.614492 5988 scope.go:122] "RemoveContainer" containerID="338c614bd6cf54300751de1dab2a927561bfbf6613891095f32750fe58b175a2" Nov 03 19:27:46.625314 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:46.625290 5988 scope.go:122] "RemoveContainer" containerID="cfd47c64fdd6de04fca92fb2d5b01db597f6e8fdc2c0eb2acb28dc8ff39a4237" Nov 03 19:27:46.746252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:46.746218 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:27:46.746462 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:46.746273 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 19:27:46.789828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:46.789789 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 19:27:46.789999 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:46.789860 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 19:27:47.746185 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:47.746138 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 19:27:47.746588 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:47.746210 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 19:27:47.789654 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:47.789614 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 19:27:47.789845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:47.789678 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 19:27:48.752326 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:48.752275 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:48.753060 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:48.753030 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:48.754324 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:48.754289 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" Nov 03 19:27:48.855829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:48.855795 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:48.856298 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:48.856092 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:48.857036 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:48.857014 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" Nov 03 19:27:53.175205 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:53.175162 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:55.937493 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:27:55.937456 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:27:57.030305 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:57.030260 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:58.038362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:58.038326 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:27:59.506208 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:27:59.506167 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:28:25.937069 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:28:25.936981 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:28:31.044567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:31.044521 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4"] Nov 03 19:28:31.091193 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:31.091158 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421c7e4e-hvpp4"] Nov 03 19:28:31.271379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:31.271335 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963119f8-95da-43ea-85c5-65e2d89c2596" path="/var/lib/kubelet/pods/963119f8-95da-43ea-85c5-65e2d89c2596/volumes" Nov 03 19:28:44.043960 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:44.043924 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg"] Nov 03 19:28:44.090281 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:44.090253 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8600c1f2-2r5hg"] Nov 03 19:28:44.713532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:44.713498 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:28:45.273916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:45.273883 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e377303f-5708-4b1e-b3a4-43faaebe237f" path="/var/lib/kubelet/pods/e377303f-5708-4b1e-b3a4-43faaebe237f/volumes" Nov 03 19:28:46.674491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:46.674438 5988 scope.go:122] "RemoveContainer" containerID="f1c1cff0b360069fabe25999a6a0e7440926b97defafd79d2d99eec85bdfc4ac" Nov 03 19:28:54.417971 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.417936 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss"] Nov 03 19:28:54.422764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.422744 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.508478 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508447 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.508647 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508486 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.508647 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508609 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.508824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508658 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.508824 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508794 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.508925 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508843 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.508925 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508869 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfvh\" (UniqueName: \"kubernetes.io/projected/daf80397-1f38-4bb7-ac99-3a6ac25116b3-kube-api-access-mgfvh\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.509031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508953 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.509031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.508996 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.520831 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.520807 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss"] Nov 03 19:28:54.610259 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610223 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610266 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610307 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610326 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610343 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfvh\" (UniqueName: \"kubernetes.io/projected/daf80397-1f38-4bb7-ac99-3a6ac25116b3-kube-api-access-mgfvh\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610384 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610419 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610801 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610456 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610801 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610484 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610801 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610743 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.610801 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610783 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.611015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610848 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.611015 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.610881 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.611290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.611256 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.612859 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.612835 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.613006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.612991 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.668445 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.668376 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf80397-1f38-4bb7-ac99-3a6ac25116b3-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.668445 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.668397 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfvh\" (UniqueName: \"kubernetes.io/projected/daf80397-1f38-4bb7-ac99-3a6ac25116b3-kube-api-access-mgfvh\") pod \"maas-default-gateway-openshift-default-7854949c9-jr2ss\" (UID: \"daf80397-1f38-4bb7-ac99-3a6ac25116b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:54.736372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:54.736332 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:55.153094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.152958 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss"] Nov 03 19:28:55.156024 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:28:55.155992 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf80397_1f38_4bb7_ac99_3a6ac25116b3.slice/crio-65c09ad40b1c55eb265e807dae43d5d757d1aa9b7272788c4b3c9a52a302ecc0 WatchSource:0}: Error finding container 65c09ad40b1c55eb265e807dae43d5d757d1aa9b7272788c4b3c9a52a302ecc0: Status 404 returned error can't find the container with id 65c09ad40b1c55eb265e807dae43d5d757d1aa9b7272788c4b3c9a52a302ecc0 Nov 03 19:28:55.158327 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.158296 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:28:55.158419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.158358 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:28:55.158419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.158386 5988 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 19:28:55.394157 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.394109 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" event={"ID":"daf80397-1f38-4bb7-ac99-3a6ac25116b3","Type":"ContainerStarted","Data":"fbda64c76064aee6e3e63bf21797cb8d39c7956b467cdbb715b1ac6a8113c9ca"} Nov 03 19:28:55.394352 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.394186 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" event={"ID":"daf80397-1f38-4bb7-ac99-3a6ac25116b3","Type":"ContainerStarted","Data":"65c09ad40b1c55eb265e807dae43d5d757d1aa9b7272788c4b3c9a52a302ecc0"} Nov 03 19:28:55.509973 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.509767 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" podStartSLOduration=1.509750775 podStartE2EDuration="1.509750775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:28:55.509593912 +0000 UTC m=+3010.391010017" watchObservedRunningTime="2026-11-03 19:28:55.509750775 +0000 UTC m=+3010.391166879" Nov 03 19:28:55.736537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:55.736488 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:28:55.935915 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:28:55.935877 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:28:56.738779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:56.738747 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jr2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 19:28:56.739220 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:56.738804 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" podUID="daf80397-1f38-4bb7-ac99-3a6ac25116b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 19:28:57.736942 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:57.736907 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jr2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 19:28:57.737210 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:57.736957 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" podUID="daf80397-1f38-4bb7-ac99-3a6ac25116b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 19:28:58.409486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:58.409453 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:28:58.737743 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:58.737644 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jr2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 19:28:58.737743 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:58.737716 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" podUID="daf80397-1f38-4bb7-ac99-3a6ac25116b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 19:28:59.737446 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:59.737408 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jr2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 19:28:59.737882 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:28:59.737480 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" podUID="daf80397-1f38-4bb7-ac99-3a6ac25116b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 19:29:00.737810 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:00.737773 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jr2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 19:29:00.738225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:00.737827 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" podUID="daf80397-1f38-4bb7-ac99-3a6ac25116b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 19:29:01.737450 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:01.737410 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-jr2ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 03 19:29:01.737634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:01.737478 5988 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" podUID="daf80397-1f38-4bb7-ac99-3a6ac25116b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 03 19:29:03.348851 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:03.348819 5988 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:29:03.349376 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:03.349105 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:29:03.350234 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:03.350210 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-jr2ss" Nov 03 19:29:06.117463 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:06.117425 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:29:17.535237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:17.535202 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:29:25.935983 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:29:25.935941 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:29:37.579994 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:37.579955 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:29:46.707247 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:46.707214 5988 scope.go:122] "RemoveContainer" containerID="07cfb99bae58896051cbd0290e9018f058b8b5070cfe5ff924bfafeb47530b25" Nov 03 19:29:46.716308 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:46.716284 5988 scope.go:122] "RemoveContainer" containerID="9028acb53d27b679a22687fc70f79a364bb066f0ec12711aebdd5b17e4d3a9b3" Nov 03 19:29:55.936635 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:29:55.936553 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:29:56.186921 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:56.186826 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:29:58.839395 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:29:58.839356 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:30:12.143496 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:12.143455 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:30:13.265489 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:13.265449 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:30:15.308895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.308859 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n"] Nov 03 19:30:15.312236 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.312218 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.361819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.361784 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:30:15.408313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.408282 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:30:15.409703 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.409655 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n"] Nov 03 19:30:15.477041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.477002 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.477271 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.477069 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.477271 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.477217 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgxwj\" (UniqueName: \"kubernetes.io/projected/38d174c4-bbf5-4ed0-9da4-092af02772ed-kube-api-access-hgxwj\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.577840 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.577802 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.578054 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.577871 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgxwj\" (UniqueName: \"kubernetes.io/projected/38d174c4-bbf5-4ed0-9da4-092af02772ed-kube-api-access-hgxwj\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.578054 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.577927 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.578565 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.578543 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.580332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.580306 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.636852 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.636817 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgxwj\" (UniqueName: \"kubernetes.io/projected/38d174c4-bbf5-4ed0-9da4-092af02772ed-kube-api-access-hgxwj\") pod \"maas-api-revoke-sub-046e1343-jw29n\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:15.924752 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:15.924637 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:16.292843 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:16.292660 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n"] Nov 03 19:30:16.295811 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:30:16.295775 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d174c4_bbf5_4ed0_9da4_092af02772ed.slice/crio-c15ab929f7727aab67066193a17577148f480d2d4cc5bbb61eb6e23feedfc30e WatchSource:0}: Error finding container c15ab929f7727aab67066193a17577148f480d2d4cc5bbb61eb6e23feedfc30e: Status 404 returned error can't find the container with id c15ab929f7727aab67066193a17577148f480d2d4cc5bbb61eb6e23feedfc30e Nov 03 19:30:16.831712 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:16.831665 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" event={"ID":"38d174c4-bbf5-4ed0-9da4-092af02772ed","Type":"ContainerStarted","Data":"c007a860a5ae6ddc8ffa3f63c479d9a1a9b4600c832dc5e1251abb0f887226af"} Nov 03 19:30:16.832140 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:16.831745 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" event={"ID":"38d174c4-bbf5-4ed0-9da4-092af02772ed","Type":"ContainerStarted","Data":"c15ab929f7727aab67066193a17577148f480d2d4cc5bbb61eb6e23feedfc30e"} Nov 03 19:30:16.940603 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:16.940551 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" podStartSLOduration=1.940537052 podStartE2EDuration="1.940537052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:16.940313084 +0000 UTC m=+3091.821729187" watchObservedRunningTime="2026-11-03 19:30:16.940537052 +0000 UTC m=+3091.821953153" Nov 03 19:30:17.838427 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:17.838394 5988 generic.go:356] "Generic (PLEG): container finished" podID="38d174c4-bbf5-4ed0-9da4-092af02772ed" containerID="c007a860a5ae6ddc8ffa3f63c479d9a1a9b4600c832dc5e1251abb0f887226af" exitCode=0 Nov 03 19:30:17.838917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:17.838481 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" event={"ID":"38d174c4-bbf5-4ed0-9da4-092af02772ed","Type":"ContainerDied","Data":"c007a860a5ae6ddc8ffa3f63c479d9a1a9b4600c832dc5e1251abb0f887226af"} Nov 03 19:30:18.984372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:18.984344 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:19.115137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.115045 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d174c4-bbf5-4ed0-9da4-092af02772ed-kube-api-access-hgxwj\" (UniqueName: \"kubernetes.io/projected/38d174c4-bbf5-4ed0-9da4-092af02772ed-kube-api-access-hgxwj\") pod \"38d174c4-bbf5-4ed0-9da4-092af02772ed\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " Nov 03 19:30:19.115137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.115093 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-openshift-ca\") pod \"38d174c4-bbf5-4ed0-9da4-092af02772ed\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " Nov 03 19:30:19.115137 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.115127 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-xks-ca\") pod \"38d174c4-bbf5-4ed0-9da4-092af02772ed\" (UID: \"38d174c4-bbf5-4ed0-9da4-092af02772ed\") " Nov 03 19:30:19.115934 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.115897 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-openshift-ca" pod "38d174c4-bbf5-4ed0-9da4-092af02772ed" (UID: "38d174c4-bbf5-4ed0-9da4-092af02772ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:30:19.117598 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.117567 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-xks-ca" pod "38d174c4-bbf5-4ed0-9da4-092af02772ed" (UID: "38d174c4-bbf5-4ed0-9da4-092af02772ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:30:19.118864 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.118811 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d174c4-bbf5-4ed0-9da4-092af02772ed-kube-api-access-hgxwj" pod "38d174c4-bbf5-4ed0-9da4-092af02772ed" (UID: "38d174c4-bbf5-4ed0-9da4-092af02772ed"). InnerVolumeSpecName "kube-api-access-hgxwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:30:19.216243 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.216191 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgxwj\" (UniqueName: \"kubernetes.io/projected/38d174c4-bbf5-4ed0-9da4-092af02772ed-kube-api-access-hgxwj\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.216243 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.216237 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.216243 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.216249 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d174c4-bbf5-4ed0-9da4-092af02772ed-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:30:19.853714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.853674 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" Nov 03 19:30:19.853714 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.853674 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n" event={"ID":"38d174c4-bbf5-4ed0-9da4-092af02772ed","Type":"ContainerDied","Data":"c15ab929f7727aab67066193a17577148f480d2d4cc5bbb61eb6e23feedfc30e"} Nov 03 19:30:19.853961 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:19.853742 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15ab929f7727aab67066193a17577148f480d2d4cc5bbb61eb6e23feedfc30e" Nov 03 19:30:25.936991 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:30:25.936953 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:30:28.185513 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:28.185469 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:30:39.271564 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:39.271528 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:30:45.046252 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:45.046201 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr"] Nov 03 19:30:45.092551 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:45.092514 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd8fec9d-d8xnr"] Nov 03 19:30:45.322767 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:45.322675 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88eead85-009a-4b4a-9f64-f22166adcccc" path="/var/lib/kubelet/pods/88eead85-009a-4b4a-9f64-f22166adcccc/volumes" Nov 03 19:30:46.739537 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:46.739498 5988 scope.go:122] "RemoveContainer" containerID="578f95ae15274272d04e7e47480d10cee78e7144ebf08577d9bcbff757d81203" Nov 03 19:30:46.781133 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:46.781087 5988 scope.go:122] "RemoveContainer" containerID="662b881a44a634cecf0043594c449aceedb36f30802c955f6c77aac4b49e3bea" Nov 03 19:30:50.387042 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:50.387002 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:30:55.936566 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:30:55.936525 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:30:59.208853 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:59.208814 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:30:59.410582 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:30:59.410534 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:31:13.595823 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:13.595779 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:31:23.593724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:23.593668 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:31:25.937521 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:31:25.937474 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:31:42.807372 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:42.807326 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:31:43.053504 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:43.053469 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:31:44.773826 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:44.773791 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:31:46.838900 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:46.838868 5988 scope.go:122] "RemoveContainer" containerID="16ded19dce8f2aba7017deede7a3ac18dd21c95e499eff80ef3aeb360ff34f7d" Nov 03 19:31:46.847661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:46.847640 5988 scope.go:122] "RemoveContainer" containerID="16a39ce472063d2b891382a355b7095bbdf7658d3f4d256b01e5faaebe99530a" Nov 03 19:31:46.855547 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:46.855526 5988 scope.go:122] "RemoveContainer" containerID="d45555f77bb871287fde07953b3a9f3405478fd518d43cc3a50a768e4f5ac15d" Nov 03 19:31:55.936369 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:31:55.936323 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:31:57.781812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:57.781776 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:31:59.563065 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:31:59.563030 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:32:17.271567 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:17.271528 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:32:19.796709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:19.795787 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-hvlvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:32:19.796709 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:19.795861 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-hvlvx" podUID="71191280-54d7-4559-81da-1996a3718675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:32:19.907836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:19.907793 5988 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q7v6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:32:19.908149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:19.908112 5988 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q7v6d" podUID="54551a8b-0cf3-4824-b9ab-6e8858ec5afe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:32:25.936661 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:32:25.936626 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:32:39.168846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:39.168808 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:32:52.271692 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:52.271642 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:32:55.935659 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:32:55.935582 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:32:59.937108 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:32:59.937069 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:33:03.896547 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:33:03.896510 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:33:25.936312 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:33:25.936270 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:33:25.995823 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:33:25.995793 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:33:36.298599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:33:36.298565 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:33:51.477612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:33:51.477570 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:33:55.935979 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:33:55.935944 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:34:00.302041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:34:00.302004 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:34:02.127050 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:34:02.127010 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:34:14.488828 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:34:14.488794 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:34:22.715546 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:34:22.715514 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:34:25.496830 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:34:25.496745 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:34:25.936486 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:34:25.936449 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:34:45.995578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:34:45.995535 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:34:55.936792 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:34:55.936748 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:34:56.905155 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:34:56.905115 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:35:00.708094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:00.708014 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:03.143815 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:03.143782 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:03.739306 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:03.739227 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:03.949302 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:03.949243 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:08.410421 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:08.410369 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:08.615811 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:08.615765 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:11.655711 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:11.655649 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:11.860495 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:11.860443 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:14.583555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:14.583501 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:14.806926 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:14.806868 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:15.018575 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:15.018469 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:15.359439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:15.359402 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:19.045991 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:19.045953 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n"] Nov 03 19:35:19.092435 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:19.092397 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jw29n"] Nov 03 19:35:19.209005 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:19.208964 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:19.272160 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:19.272129 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d174c4-bbf5-4ed0-9da4-092af02772ed" path="/var/lib/kubelet/pods/38d174c4-bbf5-4ed0-9da4-092af02772ed/volumes" Nov 03 19:35:22.396082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:22.396041 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:35:23.865727 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:23.865676 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:25.936672 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:35:25.936634 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:35:28.456771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:28.456733 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:33.287845 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:33.287810 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:33.508149 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:33.508120 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:33.794610 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:33.794579 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:34.001034 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:34.000993 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:38.215389 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:38.215337 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:38.419181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:38.419144 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:42.803313 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:42.803257 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:43.114020 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:43.113976 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:35:47.395604 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:47.395564 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:51.939332 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:51.939295 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:35:55.936497 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:35:55.936418 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:35:56.525664 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:35:56.525621 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:01.038070 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:01.038028 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:01.239235 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:01.239202 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:05.590417 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:05.590378 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:09.455963 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:09.455922 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:36:10.107494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:10.107455 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:14.703817 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:14.703782 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:19.308895 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:19.308852 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:23.508439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.508394 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c"] Nov 03 19:36:23.509042 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.509024 5988 cpu_manager.go:414] "containerMap: removing container" podUID="38d174c4-bbf5-4ed0-9da4-092af02772ed" containerName="revoke-keys" Nov 03 19:36:23.509094 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.509045 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d174c4-bbf5-4ed0-9da4-092af02772ed" containerName="revoke-keys" Nov 03 19:36:23.509188 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.509176 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d174c4-bbf5-4ed0-9da4-092af02772ed" containerName="revoke-keys" Nov 03 19:36:23.512916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.512891 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.566412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.566376 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c"] Nov 03 19:36:23.569168 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.569133 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.569341 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.569263 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czc8p\" (UniqueName: \"kubernetes.io/projected/66d21818-ba3f-4358-91f3-da3ff27de9d3-kube-api-access-czc8p\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.569407 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.569312 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.609016 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.608983 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:36:23.609016 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.608992 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:36:23.670513 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.670469 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.670513 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.670558 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czc8p\" (UniqueName: \"kubernetes.io/projected/66d21818-ba3f-4358-91f3-da3ff27de9d3-kube-api-access-czc8p\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.670877 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.670592 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.671203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.671182 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.673068 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.673041 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.728368 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.728336 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czc8p\" (UniqueName: \"kubernetes.io/projected/66d21818-ba3f-4358-91f3-da3ff27de9d3-kube-api-access-czc8p\") pod \"maas-api-revoke-sub-960ff1f6-xmx9c\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:23.824343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.824309 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:23.826181 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:23.826161 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:24.195635 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:24.195605 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c"] Nov 03 19:36:24.198021 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:36:24.197985 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d21818_ba3f_4358_91f3_da3ff27de9d3.slice/crio-4c173c9f459677885a488feb2ae2dd2cfc45640aef04ed6a18441d2fb3d2caf6 WatchSource:0}: Error finding container 4c173c9f459677885a488feb2ae2dd2cfc45640aef04ed6a18441d2fb3d2caf6: Status 404 returned error can't find the container with id 4c173c9f459677885a488feb2ae2dd2cfc45640aef04ed6a18441d2fb3d2caf6 Nov 03 19:36:24.860369 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:24.860328 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" event={"ID":"66d21818-ba3f-4358-91f3-da3ff27de9d3","Type":"ContainerStarted","Data":"9f868dceced58594bc9b013cd9853a8dc8d4344ea88134815bfd43731e41b885"} Nov 03 19:36:24.860889 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:24.860394 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" event={"ID":"66d21818-ba3f-4358-91f3-da3ff27de9d3","Type":"ContainerStarted","Data":"4c173c9f459677885a488feb2ae2dd2cfc45640aef04ed6a18441d2fb3d2caf6"} Nov 03 19:36:24.970203 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:24.970147 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" podStartSLOduration=1.970131324 podStartE2EDuration="1.970131324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:24.969120512 +0000 UTC m=+3459.850536616" watchObservedRunningTime="2026-11-03 19:36:24.970131324 +0000 UTC m=+3459.851547426" Nov 03 19:36:25.866619 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:25.866582 5988 generic.go:356] "Generic (PLEG): container finished" podID="66d21818-ba3f-4358-91f3-da3ff27de9d3" containerID="9f868dceced58594bc9b013cd9853a8dc8d4344ea88134815bfd43731e41b885" exitCode=0 Nov 03 19:36:25.867042 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:25.866668 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" event={"ID":"66d21818-ba3f-4358-91f3-da3ff27de9d3","Type":"ContainerDied","Data":"9f868dceced58594bc9b013cd9853a8dc8d4344ea88134815bfd43731e41b885"} Nov 03 19:36:25.936337 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:36:25.936304 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:36:27.008779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.008754 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:27.106032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.105986 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-xks-ca\") pod \"66d21818-ba3f-4358-91f3-da3ff27de9d3\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " Nov 03 19:36:27.106223 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.106059 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-openshift-ca\") pod \"66d21818-ba3f-4358-91f3-da3ff27de9d3\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " Nov 03 19:36:27.106223 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.106108 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d21818-ba3f-4358-91f3-da3ff27de9d3-kube-api-access-czc8p\" (UniqueName: \"kubernetes.io/projected/66d21818-ba3f-4358-91f3-da3ff27de9d3-kube-api-access-czc8p\") pod \"66d21818-ba3f-4358-91f3-da3ff27de9d3\" (UID: \"66d21818-ba3f-4358-91f3-da3ff27de9d3\") " Nov 03 19:36:27.106394 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.106369 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-openshift-ca" pod "66d21818-ba3f-4358-91f3-da3ff27de9d3" (UID: "66d21818-ba3f-4358-91f3-da3ff27de9d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:36:27.106515 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.106500 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:27.108235 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.108200 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-xks-ca" pod "66d21818-ba3f-4358-91f3-da3ff27de9d3" (UID: "66d21818-ba3f-4358-91f3-da3ff27de9d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:27.108360 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.108277 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d21818-ba3f-4358-91f3-da3ff27de9d3-kube-api-access-czc8p" pod "66d21818-ba3f-4358-91f3-da3ff27de9d3" (UID: "66d21818-ba3f-4358-91f3-da3ff27de9d3"). InnerVolumeSpecName "kube-api-access-czc8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:27.207813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.207730 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czc8p\" (UniqueName: \"kubernetes.io/projected/66d21818-ba3f-4358-91f3-da3ff27de9d3-kube-api-access-czc8p\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:27.207813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.207760 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d21818-ba3f-4358-91f3-da3ff27de9d3-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:27.877723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.877666 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" Nov 03 19:36:27.877723 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.877706 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c" event={"ID":"66d21818-ba3f-4358-91f3-da3ff27de9d3","Type":"ContainerDied","Data":"4c173c9f459677885a488feb2ae2dd2cfc45640aef04ed6a18441d2fb3d2caf6"} Nov 03 19:36:27.877946 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:27.877757 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c173c9f459677885a488feb2ae2dd2cfc45640aef04ed6a18441d2fb3d2caf6" Nov 03 19:36:28.400493 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:28.400460 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:32.903902 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:32.903864 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:35.271004 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.270968 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:35.665366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.665329 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs"] Nov 03 19:36:35.665772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.665758 5988 cpu_manager.go:414] "containerMap: removing container" podUID="66d21818-ba3f-4358-91f3-da3ff27de9d3" containerName="revoke-keys" Nov 03 19:36:35.665820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.665772 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d21818-ba3f-4358-91f3-da3ff27de9d3" containerName="revoke-keys" Nov 03 19:36:35.665867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.665857 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d21818-ba3f-4358-91f3-da3ff27de9d3" containerName="revoke-keys" Nov 03 19:36:35.669256 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.669236 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.718248 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.718214 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:36:35.764785 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.764740 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:36:35.766127 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.766100 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs"] Nov 03 19:36:35.789581 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.789545 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.789790 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.789588 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6p8h\" (UniqueName: \"kubernetes.io/projected/c27f8428-bcb1-4ccf-b085-0743a72b8c59-kube-api-access-q6p8h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.789790 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.789612 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.890995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.890960 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.890995 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.891005 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6p8h\" (UniqueName: \"kubernetes.io/projected/c27f8428-bcb1-4ccf-b085-0743a72b8c59-kube-api-access-q6p8h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.891273 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.891099 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.891631 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.891604 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.893484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.893460 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.948731 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.948633 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6p8h\" (UniqueName: \"kubernetes.io/projected/c27f8428-bcb1-4ccf-b085-0743a72b8c59-kube-api-access-q6p8h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:35.982461 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:35.982425 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:36.347540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:36.347467 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs"] Nov 03 19:36:36.350945 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:36:36.350914 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27f8428_bcb1_4ccf_b085_0743a72b8c59.slice/crio-e9ff3f18c302da0ce90141de5f87970370f36316d60017b0e4f3bc6e29b0541e WatchSource:0}: Error finding container e9ff3f18c302da0ce90141de5f87970370f36316d60017b0e4f3bc6e29b0541e: Status 404 returned error can't find the container with id e9ff3f18c302da0ce90141de5f87970370f36316d60017b0e4f3bc6e29b0541e Nov 03 19:36:36.929441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:36.929348 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" event={"ID":"c27f8428-bcb1-4ccf-b085-0743a72b8c59","Type":"ContainerStarted","Data":"a8a54d2673b8946f3838070cfe767824b150f01f8daa4ce0b1a15b301a957e8e"} Nov 03 19:36:36.929441 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:36.929416 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" event={"ID":"c27f8428-bcb1-4ccf-b085-0743a72b8c59","Type":"ContainerStarted","Data":"e9ff3f18c302da0ce90141de5f87970370f36316d60017b0e4f3bc6e29b0541e"} Nov 03 19:36:37.038491 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:37.038436 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" podStartSLOduration=2.038422081 podStartE2EDuration="2.038422081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:37.037832572 +0000 UTC m=+3471.919248686" watchObservedRunningTime="2026-11-03 19:36:37.038422081 +0000 UTC m=+3471.919838179" Nov 03 19:36:37.505263 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:37.505182 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:37.935916 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:37.935878 5988 generic.go:356] "Generic (PLEG): container finished" podID="c27f8428-bcb1-4ccf-b085-0743a72b8c59" containerID="a8a54d2673b8946f3838070cfe767824b150f01f8daa4ce0b1a15b301a957e8e" exitCode=0 Nov 03 19:36:37.936096 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:37.935964 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" event={"ID":"c27f8428-bcb1-4ccf-b085-0743a72b8c59","Type":"ContainerDied","Data":"a8a54d2673b8946f3838070cfe767824b150f01f8daa4ce0b1a15b301a957e8e"} Nov 03 19:36:38.386479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:38.386446 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:36:39.088679 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.088653 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:39.224415 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.224315 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27f8428-bcb1-4ccf-b085-0743a72b8c59-kube-api-access-q6p8h\" (UniqueName: \"kubernetes.io/projected/c27f8428-bcb1-4ccf-b085-0743a72b8c59-kube-api-access-q6p8h\") pod \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " Nov 03 19:36:39.224602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.224443 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-xks-ca\") pod \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " Nov 03 19:36:39.224602 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.224475 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-openshift-ca\") pod \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\" (UID: \"c27f8428-bcb1-4ccf-b085-0743a72b8c59\") " Nov 03 19:36:39.224936 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.224905 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-openshift-ca" pod "c27f8428-bcb1-4ccf-b085-0743a72b8c59" (UID: "c27f8428-bcb1-4ccf-b085-0743a72b8c59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:36:39.226657 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.226632 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-xks-ca" pod "c27f8428-bcb1-4ccf-b085-0743a72b8c59" (UID: "c27f8428-bcb1-4ccf-b085-0743a72b8c59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:36:39.226791 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.226670 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27f8428-bcb1-4ccf-b085-0743a72b8c59-kube-api-access-q6p8h" pod "c27f8428-bcb1-4ccf-b085-0743a72b8c59" (UID: "c27f8428-bcb1-4ccf-b085-0743a72b8c59"). InnerVolumeSpecName "kube-api-access-q6p8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:36:39.325460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.325426 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6p8h\" (UniqueName: \"kubernetes.io/projected/c27f8428-bcb1-4ccf-b085-0743a72b8c59-kube-api-access-q6p8h\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:39.325460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.325459 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:39.325460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.325469 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c27f8428-bcb1-4ccf-b085-0743a72b8c59-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:36:39.949822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.949786 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" event={"ID":"c27f8428-bcb1-4ccf-b085-0743a72b8c59","Type":"ContainerDied","Data":"e9ff3f18c302da0ce90141de5f87970370f36316d60017b0e4f3bc6e29b0541e"} Nov 03 19:36:39.949822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.949814 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs" Nov 03 19:36:39.950041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:39.949843 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9ff3f18c302da0ce90141de5f87970370f36316d60017b0e4f3bc6e29b0541e" Nov 03 19:36:40.398634 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:40.398595 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:42.430271 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:42.430237 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:46.377939 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:46.377900 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:46.857237 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:46.857198 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:46.930822 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:46.930791 5988 scope.go:122] "RemoveContainer" containerID="c007a860a5ae6ddc8ffa3f63c479d9a1a9b4600c832dc5e1251abb0f887226af" Nov 03 19:36:47.058753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:47.058652 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:51.009977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:51.009941 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:36:51.427035 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:51.426993 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:51.686553 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:51.686464 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:51.944000 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:51.943910 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs"] Nov 03 19:36:51.990289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:51.990261 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-020814-qpbjs"] Nov 03 19:36:52.435299 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:52.435257 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:53.271771 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:53.271738 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27f8428-bcb1-4ccf-b085-0743a72b8c59" path="/var/lib/kubelet/pods/c27f8428-bcb1-4ccf-b085-0743a72b8c59/volumes" Nov 03 19:36:55.935230 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:36:55.935196 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:36:58.373528 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:58.373496 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:58.576830 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:58.576795 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:36:58.778362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:36:58.778280 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:01.359115 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:01.359073 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:15.098656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:15.098625 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:37:16.765799 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:16.765760 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:17.520379 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:17.520304 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:23.065724 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:23.065674 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:25.937040 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:37:25.936961 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:37:26.197337 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:26.197261 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:37:29.203695 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:29.203647 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:55.094277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:55.094238 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:55.315745 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:37:55.315677 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:37:55.937592 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:37:55.937553 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:38:01.730764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:38:01.730720 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:38:25.935104 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:38:25.935066 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:38:55.937623 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:38:55.937530 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:38:59.803099 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:38:59.803062 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:39:02.176710 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:02.173089 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:16.999528 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:16.999487 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:39:23.735552 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:23.735503 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:23.945871 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:23.945835 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:25.935579 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:39:25.935540 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:39:26.703829 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:26.703795 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:39:45.769576 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:45.769540 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:46.091310 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:46.091268 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:39:46.416087 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:46.415998 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:46.680985 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:46.680892 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:50.726597 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:50.726523 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:51.108412 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:51.108376 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:54.073906 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:54.073870 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:54.276471 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:54.276435 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:55.939474 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:39:55.939427 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:39:57.430142 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:57.430108 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:57.644959 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:57.644926 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:39:57.856540 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:39:57.856506 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:02.564062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:02.564022 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:02.766311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:02.766275 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:05.587370 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:05.587330 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:40:07.707607 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:07.707563 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:12.864913 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:12.864865 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:16.832049 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:16.832016 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:17.057192 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:17.057155 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:18.285754 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:18.285713 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:19.760868 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:19.760826 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:23.345082 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:23.345045 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:24.301799 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:24.301762 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:40:25.936431 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:40:25.936388 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:40:28.446923 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:28.446887 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:33.455063 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:33.455020 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:38.444772 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:38.444731 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:43.457311 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:43.457260 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:48.451958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:48.451909 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:49.980292 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:49.980262 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:40:53.443475 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:53.443435 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:40:55.934834 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:40:55.934799 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:40:58.448498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:40:58.448461 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:02.869029 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:02.868992 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:03.452242 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:03.452208 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:04.992483 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:04.992450 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t"] Nov 03 19:41:04.992950 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:04.992934 5988 cpu_manager.go:414] "containerMap: removing container" podUID="c27f8428-bcb1-4ccf-b085-0743a72b8c59" containerName="revoke-keys" Nov 03 19:41:04.992996 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:04.992951 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27f8428-bcb1-4ccf-b085-0743a72b8c59" containerName="revoke-keys" Nov 03 19:41:04.993049 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:04.993041 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27f8428-bcb1-4ccf-b085-0743a72b8c59" containerName="revoke-keys" Nov 03 19:41:04.996194 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:04.996173 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.045347 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.045302 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:41:05.045347 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.045301 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:41:05.093009 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.092980 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t"] Nov 03 19:41:05.115131 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.115100 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mr2\" (UniqueName: \"kubernetes.io/projected/6637b2f0-c203-47d9-9227-3542eab20213-kube-api-access-q8mr2\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.115325 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.115236 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.115325 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.115297 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.216663 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.216621 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.216663 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.216671 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.216958 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.216758 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mr2\" (UniqueName: \"kubernetes.io/projected/6637b2f0-c203-47d9-9227-3542eab20213-kube-api-access-q8mr2\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.217469 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.217441 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.219010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.218992 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.274357 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.274284 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mr2\" (UniqueName: \"kubernetes.io/projected/6637b2f0-c203-47d9-9227-3542eab20213-kube-api-access-q8mr2\") pod \"maas-api-revoke-sub-82a86703-cvn8t\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.310143 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.310107 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:05.676541 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:05.676518 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t"] Nov 03 19:41:05.679493 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:41:05.679461 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6637b2f0_c203_47d9_9227_3542eab20213.slice/crio-b156212cb8bccadef03afcd1960c8871329381617ecd31a8841154de958703e4 WatchSource:0}: Error finding container b156212cb8bccadef03afcd1960c8871329381617ecd31a8841154de958703e4: Status 404 returned error can't find the container with id b156212cb8bccadef03afcd1960c8871329381617ecd31a8841154de958703e4 Nov 03 19:41:06.237273 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:06.237185 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" event={"ID":"6637b2f0-c203-47d9-9227-3542eab20213","Type":"ContainerStarted","Data":"e55fdfb07d356a7962d6826003eeaeb6f7a27ba93bc05d2d0cd1b5ddf8c596c0"} Nov 03 19:41:06.237273 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:06.237239 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" event={"ID":"6637b2f0-c203-47d9-9227-3542eab20213","Type":"ContainerStarted","Data":"b156212cb8bccadef03afcd1960c8871329381617ecd31a8841154de958703e4"} Nov 03 19:41:06.345361 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:06.345297 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" podStartSLOduration=2.345279742 podStartE2EDuration="2.345279742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:06.345022482 +0000 UTC m=+3741.226438584" watchObservedRunningTime="2026-11-03 19:41:06.345279742 +0000 UTC m=+3741.226695844" Nov 03 19:41:07.243115 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:07.243077 5988 generic.go:356] "Generic (PLEG): container finished" podID="6637b2f0-c203-47d9-9227-3542eab20213" containerID="e55fdfb07d356a7962d6826003eeaeb6f7a27ba93bc05d2d0cd1b5ddf8c596c0" exitCode=0 Nov 03 19:41:07.243548 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:07.243168 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" event={"ID":"6637b2f0-c203-47d9-9227-3542eab20213","Type":"ContainerDied","Data":"e55fdfb07d356a7962d6826003eeaeb6f7a27ba93bc05d2d0cd1b5ddf8c596c0"} Nov 03 19:41:08.389922 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.389893 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:08.446867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.446832 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6637b2f0-c203-47d9-9227-3542eab20213-kube-api-access-q8mr2\" (UniqueName: \"kubernetes.io/projected/6637b2f0-c203-47d9-9227-3542eab20213-kube-api-access-q8mr2\") pod \"6637b2f0-c203-47d9-9227-3542eab20213\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " Nov 03 19:41:08.447046 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.446888 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-openshift-ca\") pod \"6637b2f0-c203-47d9-9227-3542eab20213\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " Nov 03 19:41:08.447046 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.446921 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-xks-ca\") pod \"6637b2f0-c203-47d9-9227-3542eab20213\" (UID: \"6637b2f0-c203-47d9-9227-3542eab20213\") " Nov 03 19:41:08.447303 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.447279 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-openshift-ca" pod "6637b2f0-c203-47d9-9227-3542eab20213" (UID: "6637b2f0-c203-47d9-9227-3542eab20213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:08.449093 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.449062 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-xks-ca" pod "6637b2f0-c203-47d9-9227-3542eab20213" (UID: "6637b2f0-c203-47d9-9227-3542eab20213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:08.449258 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.449181 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6637b2f0-c203-47d9-9227-3542eab20213-kube-api-access-q8mr2" pod "6637b2f0-c203-47d9-9227-3542eab20213" (UID: "6637b2f0-c203-47d9-9227-3542eab20213"). InnerVolumeSpecName "kube-api-access-q8mr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:08.503238 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.503120 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:08.548037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.548002 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mr2\" (UniqueName: \"kubernetes.io/projected/6637b2f0-c203-47d9-9227-3542eab20213-kube-api-access-q8mr2\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:08.548037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.548031 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:08.548037 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:08.548040 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6637b2f0-c203-47d9-9227-3542eab20213-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:09.254956 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:09.254927 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" Nov 03 19:41:09.254956 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:09.254942 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t" event={"ID":"6637b2f0-c203-47d9-9227-3542eab20213","Type":"ContainerDied","Data":"b156212cb8bccadef03afcd1960c8871329381617ecd31a8841154de958703e4"} Nov 03 19:41:09.255180 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:09.254995 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b156212cb8bccadef03afcd1960c8871329381617ecd31a8841154de958703e4" Nov 03 19:41:13.559477 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:13.559437 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:16.189640 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.189605 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:16.629216 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.629176 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd"] Nov 03 19:41:16.629716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.629667 5988 cpu_manager.go:414] "containerMap: removing container" podUID="6637b2f0-c203-47d9-9227-3542eab20213" containerName="revoke-keys" Nov 03 19:41:16.629716 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.629707 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6637b2f0-c203-47d9-9227-3542eab20213" containerName="revoke-keys" Nov 03 19:41:16.629886 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.629862 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="6637b2f0-c203-47d9-9227-3542eab20213" containerName="revoke-keys" Nov 03 19:41:16.633299 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.633277 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.719306 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.719270 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.719516 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.719312 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fjv\" (UniqueName: \"kubernetes.io/projected/98561dec-9a06-4254-a2bf-c6261db41bc6-kube-api-access-c8fjv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.719516 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.719499 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.728644 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.728612 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 19:41:16.728644 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.728634 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fbqp5\"" Nov 03 19:41:16.730118 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.730100 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd"] Nov 03 19:41:16.820291 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.820248 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.820291 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.820290 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fjv\" (UniqueName: \"kubernetes.io/projected/98561dec-9a06-4254-a2bf-c6261db41bc6-kube-api-access-c8fjv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.820588 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.820428 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.821133 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.821112 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.822726 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.822706 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.878812 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.878773 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fjv\" (UniqueName: \"kubernetes.io/projected/98561dec-9a06-4254-a2bf-c6261db41bc6-kube-api-access-c8fjv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:16.945769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:16.945629 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:17.313568 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:17.313539 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd"] Nov 03 19:41:17.316215 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:41:17.316186 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98561dec_9a06_4254_a2bf_c6261db41bc6.slice/crio-d622195ec566c0b4aa5f3242c36883779e08b5599dba9ab88af84690daff203a WatchSource:0}: Error finding container d622195ec566c0b4aa5f3242c36883779e08b5599dba9ab88af84690daff203a: Status 404 returned error can't find the container with id d622195ec566c0b4aa5f3242c36883779e08b5599dba9ab88af84690daff203a Nov 03 19:41:18.092428 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:18.092385 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:41:18.302414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:18.302297 5988 generic.go:356] "Generic (PLEG): container finished" podID="98561dec-9a06-4254-a2bf-c6261db41bc6" containerID="af9fe91bea0b03827966118ea37953b61f6d68382a25c447e31a38f82c3577ca" exitCode=0 Nov 03 19:41:18.302607 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:18.302377 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" event={"ID":"98561dec-9a06-4254-a2bf-c6261db41bc6","Type":"ContainerDied","Data":"af9fe91bea0b03827966118ea37953b61f6d68382a25c447e31a38f82c3577ca"} Nov 03 19:41:18.302607 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:18.302490 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" event={"ID":"98561dec-9a06-4254-a2bf-c6261db41bc6","Type":"ContainerStarted","Data":"d622195ec566c0b4aa5f3242c36883779e08b5599dba9ab88af84690daff203a"} Nov 03 19:41:18.668861 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:18.668830 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:19.444077 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.444053 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:19.549599 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.549557 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-xks-ca\") pod \"98561dec-9a06-4254-a2bf-c6261db41bc6\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " Nov 03 19:41:19.549846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.549716 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98561dec-9a06-4254-a2bf-c6261db41bc6-kube-api-access-c8fjv\" (UniqueName: \"kubernetes.io/projected/98561dec-9a06-4254-a2bf-c6261db41bc6-kube-api-access-c8fjv\") pod \"98561dec-9a06-4254-a2bf-c6261db41bc6\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " Nov 03 19:41:19.549846 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.549770 5988 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-openshift-ca\") pod \"98561dec-9a06-4254-a2bf-c6261db41bc6\" (UID: \"98561dec-9a06-4254-a2bf-c6261db41bc6\") " Nov 03 19:41:19.550118 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.550093 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-openshift-ca" pod "98561dec-9a06-4254-a2bf-c6261db41bc6" (UID: "98561dec-9a06-4254-a2bf-c6261db41bc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:41:19.551907 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.551878 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-xks-ca" pod "98561dec-9a06-4254-a2bf-c6261db41bc6" (UID: "98561dec-9a06-4254-a2bf-c6261db41bc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 19:41:19.551993 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.551941 5988 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98561dec-9a06-4254-a2bf-c6261db41bc6-kube-api-access-c8fjv" pod "98561dec-9a06-4254-a2bf-c6261db41bc6" (UID: "98561dec-9a06-4254-a2bf-c6261db41bc6"). InnerVolumeSpecName "kube-api-access-c8fjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:41:19.651578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.651540 5988 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8fjv\" (UniqueName: \"kubernetes.io/projected/98561dec-9a06-4254-a2bf-c6261db41bc6-kube-api-access-c8fjv\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:19.651578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.651571 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-openshift-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:19.651578 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:19.651581 5988 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98561dec-9a06-4254-a2bf-c6261db41bc6-maas-api-xks-ca\") on node \"ip-10-0-139-161.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:41:20.312891 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:20.312854 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" event={"ID":"98561dec-9a06-4254-a2bf-c6261db41bc6","Type":"ContainerDied","Data":"d622195ec566c0b4aa5f3242c36883779e08b5599dba9ab88af84690daff203a"} Nov 03 19:41:20.313316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:20.312897 5988 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd" Nov 03 19:41:20.313316 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:20.312906 5988 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d622195ec566c0b4aa5f3242c36883779e08b5599dba9ab88af84690daff203a" Nov 03 19:41:21.209819 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:21.209781 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:23.738884 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:23.738838 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:25.361508 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:25.361466 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:25.936845 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:41:25.936809 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:41:27.045706 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:27.045651 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c"] Nov 03 19:41:27.092154 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:27.092118 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ff1f6-xmx9c"] Nov 03 19:41:27.278761 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:27.278725 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d21818-ba3f-4358-91f3-da3ff27de9d3" path="/var/lib/kubelet/pods/66d21818-ba3f-4358-91f3-da3ff27de9d3/volumes" Nov 03 19:41:28.126249 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:28.126210 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:28.621290 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:28.621254 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:28.821144 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:28.821108 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:32.706547 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:32.706509 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:32.895484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:32.895447 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:41:33.025402 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:33.025322 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:33.311720 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:33.311609 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd"] Nov 03 19:41:33.358072 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:33.358036 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-439a13-t4gcd"] Nov 03 19:41:33.802885 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:33.802853 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:35.278917 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:35.278882 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98561dec-9a06-4254-a2bf-c6261db41bc6" path="/var/lib/kubelet/pods/98561dec-9a06-4254-a2bf-c6261db41bc6/volumes" Nov 03 19:41:39.900905 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:39.900869 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:40.122789 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:40.122717 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:40.324950 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:40.324918 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:46.815032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:46.814988 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:51.298867 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:51.298827 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:41:55.935825 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:41:55.935740 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:41:58.210854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:58.210821 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:41:58.489439 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:41:58.489348 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:42:03.225097 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:42:03.225058 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:42:03.419896 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:42:03.419862 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:42:12.929635 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:42:12.929602 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:42:25.935936 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:42:25.935890 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:42:30.746174 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:42:30.746131 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:42:35.310405 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:42:35.310364 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:42:46.989490 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:42:46.989448 5988 scope.go:122] "RemoveContainer" containerID="a8a54d2673b8946f3838070cfe767824b150f01f8daa4ce0b1a15b301a957e8e" Nov 03 19:42:46.998081 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:42:46.998058 5988 scope.go:122] "RemoveContainer" containerID="9f868dceced58594bc9b013cd9853a8dc8d4344ea88134815bfd43731e41b885" Nov 03 19:42:55.935093 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:42:55.935052 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:43:03.587656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:43:03.587616 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:43:05.303088 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:43:05.303059 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:43:16.744892 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:43:16.744853 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:43:19.482926 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:43:19.482881 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:43:20.979632 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:43:20.979599 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:43:25.936443 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:43:25.936365 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:43:33.591705 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:43:33.591652 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:43:43.701732 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:43:43.701694 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:43:55.935890 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:43:55.935853 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:44:03.944596 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:03.944558 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:07.405625 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:07.405583 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:44:14.368919 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:14.368861 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:16.068557 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.068524 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8"] Nov 03 19:44:16.068989 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.068974 5988 cpu_manager.go:414] "containerMap: removing container" podUID="98561dec-9a06-4254-a2bf-c6261db41bc6" containerName="revoke-keys" Nov 03 19:44:16.069033 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.068989 5988 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98561dec-9a06-4254-a2bf-c6261db41bc6" containerName="revoke-keys" Nov 03 19:44:16.069069 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.069064 5988 memory_manager.go:365] "RemoveStaleState removing state" podUID="98561dec-9a06-4254-a2bf-c6261db41bc6" containerName="revoke-keys" Nov 03 19:44:16.072504 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.072482 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.168214 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.168183 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4r8l\"" Nov 03 19:44:16.168463 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.168445 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 19:44:16.168535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.168470 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 19:44:16.169855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.169836 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8"] Nov 03 19:44:16.211339 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.211305 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e4b483-be28-4709-bc52-38d24b72c1ff-openshift-service-ca\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.211532 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.211355 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e4b483-be28-4709-bc52-38d24b72c1ff-maas-api-tls\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.211609 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.211517 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfs59\" (UniqueName: \"kubernetes.io/projected/c3e4b483-be28-4709-bc52-38d24b72c1ff-kube-api-access-cfs59\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.211609 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.211557 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e4b483-be28-4709-bc52-38d24b72c1ff-metrics-certs\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.313091 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.313053 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e4b483-be28-4709-bc52-38d24b72c1ff-maas-api-tls\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.313294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.313150 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfs59\" (UniqueName: \"kubernetes.io/projected/c3e4b483-be28-4709-bc52-38d24b72c1ff-kube-api-access-cfs59\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.313294 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.313230 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e4b483-be28-4709-bc52-38d24b72c1ff-metrics-certs\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.313431 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.313328 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e4b483-be28-4709-bc52-38d24b72c1ff-openshift-service-ca\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.314136 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.314104 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e4b483-be28-4709-bc52-38d24b72c1ff-openshift-service-ca\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.315627 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.315600 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3e4b483-be28-4709-bc52-38d24b72c1ff-metrics-certs\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.315818 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.315799 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3e4b483-be28-4709-bc52-38d24b72c1ff-maas-api-tls\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.370720 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.370610 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfs59\" (UniqueName: \"kubernetes.io/projected/c3e4b483-be28-4709-bc52-38d24b72c1ff-kube-api-access-cfs59\") pod \"maas-api-6696c7b56d-xjct8\" (UID: \"c3e4b483-be28-4709-bc52-38d24b72c1ff\") " pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.385274 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.385245 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:16.752093 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.752023 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8"] Nov 03 19:44:16.754562 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:44:16.754532 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e4b483_be28_4709_bc52_38d24b72c1ff.slice/crio-5d7a9faa9c8c81c36cd2eac6efa3eadbade5366af6a656fdba81fa537f015dc4 WatchSource:0}: Error finding container 5d7a9faa9c8c81c36cd2eac6efa3eadbade5366af6a656fdba81fa537f015dc4: Status 404 returned error can't find the container with id 5d7a9faa9c8c81c36cd2eac6efa3eadbade5366af6a656fdba81fa537f015dc4 Nov 03 19:44:16.756031 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.756015 5988 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:44:16.765663 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:16.765625 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:17.284765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:17.284726 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" event={"ID":"c3e4b483-be28-4709-bc52-38d24b72c1ff","Type":"ContainerStarted","Data":"5d7a9faa9c8c81c36cd2eac6efa3eadbade5366af6a656fdba81fa537f015dc4"} Nov 03 19:44:18.292165 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:18.292130 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" event={"ID":"c3e4b483-be28-4709-bc52-38d24b72c1ff","Type":"ContainerStarted","Data":"644a36777c17c66143d290545593a804d7acb6093cfa82a242f5bfcc90890178"} Nov 03 19:44:18.292617 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:18.292254 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:18.349112 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:18.349076 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:18.404465 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:18.404376 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" podStartSLOduration=1.611409085 podStartE2EDuration="2.404357502s" totalImagesPullingTime="792.948417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:18.403698758 +0000 UTC m=+3933.285114866" watchObservedRunningTime="2026-11-03 19:44:18.404357502 +0000 UTC m=+3933.285773605" Nov 03 19:44:18.557390 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:18.557296 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:19.556926 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:19.556893 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:21.657769 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:21.657741 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:21.862453 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:21.862421 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:23.985719 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:23.984647 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:24.182977 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:24.182946 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:24.302795 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:24.302728 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6696c7b56d-xjct8" Nov 03 19:44:24.465366 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:24.465337 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:25.935575 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:44:25.935529 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:44:29.580474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:29.580441 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:32.286806 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:32.286778 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:44:34.654507 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:34.654475 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:39.750498 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:39.750466 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:42.018855 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:42.018825 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:42.241182 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:42.241149 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:45.245363 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:45.245333 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:50.456944 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:50.456910 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:55.454462 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:55.454383 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:55.936289 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:44:55.936255 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:44:57.929931 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:57.929896 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:44:58.129639 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:44:58.129606 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:00.447200 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:00.447169 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:04.342154 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:04.342123 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:04.540484 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:04.540443 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:05.467320 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:05.467289 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:06.761009 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:06.760969 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:06.961743 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:06.961699 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:10.378180 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:10.378144 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:14.802888 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:14.802852 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:16.384661 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:16.384621 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:16.671730 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:16.671637 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:19.197463 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:19.197424 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:45:21.061727 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:21.061664 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:22.316556 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:22.316505 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:22.945708 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:22.945659 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:23.190863 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:23.190830 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:25.935989 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:45:25.935945 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:45:27.368878 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:27.368843 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:27.566268 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:27.566233 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:30.667460 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:30.667428 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:30.867058 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:30.867020 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:33.914362 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:33.914324 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:34.109158 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:34.109127 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:39.051109 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:39.051069 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:42.008345 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:42.008310 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:45:44.196617 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:44.196580 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:49.249296 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:49.249260 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:51.546114 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:51.546055 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:52.187746 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:52.187713 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:54.622041 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:54.622004 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:45:55.935344 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:45:55.935306 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:45:59.676701 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:45:59.676660 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:04.628632 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:04.628597 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:04.832327 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:04.832290 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:09.053854 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:09.053815 5988 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t"] Nov 03 19:46:09.100401 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:09.100365 5988 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82a86703-cvn8t"] Nov 03 19:46:09.271651 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:09.271617 5988 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6637b2f0-c203-47d9-9227-3542eab20213" path="/var/lib/kubelet/pods/6637b2f0-c203-47d9-9227-3542eab20213/volumes" Nov 03 19:46:09.643189 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:09.643153 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:14.668006 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:14.667967 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:19.659612 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:19.659572 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:24.644486 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:24.644399 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:25.938775 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:46:25.938735 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:46:29.642459 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:29.642425 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:34.746789 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:34.746758 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:39.757095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:39.757059 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:44.780419 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:44.780379 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:49.761062 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:49.761022 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:54.311190 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:54.311159 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:54.759806 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:54.759772 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:46:55.935490 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:46:55.935442 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:46:59.748511 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:46:59.748470 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:04.757488 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:04.757453 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:04.980225 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:04.980191 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:09.753582 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:09.753546 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:15.085054 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:15.084997 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:20.155809 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:20.155776 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:24.467721 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:24.467649 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:24.669813 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:24.669778 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:25.936585 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:47:25.936545 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:47:30.661765 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:30.661727 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:34.510503 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:34.510468 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:36.080533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:36.080495 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:36.368555 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:36.368469 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:37.145558 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:37.145521 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:39.007402 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:39.007362 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:47:43.105606 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:43.105567 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:43.304189 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:43.304151 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:43.505083 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:43.504986 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:47:47.054071 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:47.054037 5988 scope.go:122] "RemoveContainer" containerID="e55fdfb07d356a7962d6826003eeaeb6f7a27ba93bc05d2d0cd1b5ddf8c596c0" Nov 03 19:47:47.065473 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:47.065449 5988 scope.go:122] "RemoveContainer" containerID="af9fe91bea0b03827966118ea37953b61f6d68382a25c447e31a38f82c3577ca" Nov 03 19:47:52.886289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:47:52.886252 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:47:55.936357 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:47:55.936271 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:48:05.340479 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:48:05.340442 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:48:05.795301 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:48:05.795210 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:48:25.935287 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:48:25.935241 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:48:26.702201 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:48:26.702171 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:48:55.935758 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:48:55.935721 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:48:59.808725 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:48:59.808693 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sclr2"] Nov 03 19:49:05.693697 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:05.693662 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:49:10.243289 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:10.243242 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jgk4x_e28965a9-7d03-49e4-b4ab-ed87baf626c4/manager/0.log" Nov 03 19:49:10.774783 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:10.774750 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7z57r_e644f9f1-4403-4823-a83a-e1822f286a97/manager/1.log" Nov 03 19:49:11.622535 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:11.622509 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5px4m_51a5ffea-2b78-499e-b3f3-556157548f4b/server/0.log" Nov 03 19:49:11.835968 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:11.835941 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8rqlm_6e6addd9-43a6-4705-9c7b-b9a3e35938a4/manager/0.log" Nov 03 19:49:12.684673 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:12.684640 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7z57r_e644f9f1-4403-4823-a83a-e1822f286a97/manager/0.log" Nov 03 19:49:13.638375 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:13.638338 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6696c7b56d-xjct8_c3e4b483-be28-4709-bc52-38d24b72c1ff/maas-api/0.log" Nov 03 19:49:14.583240 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:14.583210 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-r8xrq_f7d65397-5369-4595-baa2-84ee90ea69b2/postgres/0.log" Nov 03 19:49:17.811043 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:17.811010 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sclr2_71691d5d-452d-4976-998f-715fb0207c0f/limitador/0.log" Nov 03 19:49:19.650257 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:19.650224 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8f876b79-hw55c_707684d4-a619-4320-b02f-177056b4065b/kube-auth-proxy/0.log" Nov 03 19:49:20.683343 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:20.683269 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-hvlvx_71191280-54d7-4559-81da-1996a3718675/istio-proxy/0.log" Nov 03 19:49:20.951518 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:20.951421 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-jr2ss_daf80397-1f38-4bb7-ac99-3a6ac25116b3/istio-proxy/0.log" Nov 03 19:49:21.167457 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:21.167429 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-q7v6d_54551a8b-0cf3-4824-b9ab-6e8858ec5afe/istio-proxy/0.log" Nov 03 19:49:25.936581 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:49:25.936485 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:49:36.221304 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:36.221269 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jgk4x_e28965a9-7d03-49e4-b4ab-ed87baf626c4/manager/0.log" Nov 03 19:49:36.778139 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:36.778103 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7z57r_e644f9f1-4403-4823-a83a-e1822f286a97/manager/1.log" Nov 03 19:49:37.465119 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:37.465091 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5px4m_51a5ffea-2b78-499e-b3f3-556157548f4b/server/0.log" Nov 03 19:49:37.680105 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:37.680072 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8rqlm_6e6addd9-43a6-4705-9c7b-b9a3e35938a4/manager/0.log" Nov 03 19:49:38.887909 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:38.887876 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6696c7b56d-xjct8_c3e4b483-be28-4709-bc52-38d24b72c1ff/maas-api/0.log" Nov 03 19:49:39.897520 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:39.897490 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-r8xrq_f7d65397-5369-4595-baa2-84ee90ea69b2/postgres/0.log" Nov 03 19:49:42.695718 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:42.695664 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sclr2_71691d5d-452d-4976-998f-715fb0207c0f/limitador/0.log" Nov 03 19:49:44.330414 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:44.330381 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8f876b79-hw55c_707684d4-a619-4320-b02f-177056b4065b/kube-auth-proxy/0.log" Nov 03 19:49:45.381019 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:45.380993 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-hvlvx_71191280-54d7-4559-81da-1996a3718675/istio-proxy/0.log" Nov 03 19:49:45.678656 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:45.678573 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-jr2ss_daf80397-1f38-4bb7-ac99-3a6ac25116b3/istio-proxy/0.log" Nov 03 19:49:45.911483 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:49:45.911450 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-q7v6d_54551a8b-0cf3-4824-b9ab-6e8858ec5afe/istio-proxy/0.log" Nov 03 19:49:55.935770 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:49:55.935732 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:50:06.104865 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:50:06.104823 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:50:25.942650 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:50:25.942603 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:50:55.936610 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:50:55.936518 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:50:59.183011 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:50:59.182972 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:50:59.400085 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:50:59.400045 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:51:06.411228 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:06.411192 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:51:07.950764 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:07.950728 5988 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" limit=1024 Nov 03 19:51:08.402523 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:08.402496 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2np47_4d25cdfb-758e-47fa-adbc-8f6953a7be99/global-pull-secret-syncer/0.log" Nov 03 19:51:11.612092 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:11.612063 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5sdxh_06841536-c400-4c0f-8d46-5d8e24ba9c6d/konnectivity-agent/0.log" Nov 03 19:51:13.659440 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:13.659408 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-161.us-west-2.compute.internal_3a04c408b2b26c5648ca41d83d5e6490/haproxy/0.log" Nov 03 19:51:18.719625 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.719593 5988 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f"] Nov 03 19:51:18.723274 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.723253 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.772482 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.772449 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjsjt\"/\"kube-root-ca.crt\"" Nov 03 19:51:18.819115 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.819074 5988 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tjsjt\"/\"openshift-service-ca.crt\"" Nov 03 19:51:18.819314 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.819122 5988 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tjsjt\"/\"default-dockercfg-89plj\"" Nov 03 19:51:18.820534 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.820507 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f"] Nov 03 19:51:18.854064 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.854033 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-lib-modules\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.854255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.854084 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-proc\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.854255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.854176 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-sys\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.854255 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.854237 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ppq\" (UniqueName: \"kubernetes.io/projected/e7a0137e-a621-477f-9961-17eec756c979-kube-api-access-t4ppq\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.854425 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.854352 5988 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-podres\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955272 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955226 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-sys\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955303 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ppq\" (UniqueName: \"kubernetes.io/projected/e7a0137e-a621-477f-9961-17eec756c979-kube-api-access-t4ppq\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955343 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-sys\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955374 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-podres\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955448 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-lib-modules\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955474 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955467 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-podres\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955483 5988 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-proc\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955512 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-lib-modules\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:18.955753 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:18.955591 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e7a0137e-a621-477f-9961-17eec756c979-proc\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:19.013554 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:19.013476 5988 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ppq\" (UniqueName: \"kubernetes.io/projected/e7a0137e-a621-477f-9961-17eec756c979-kube-api-access-t4ppq\") pod \"perf-node-gather-daemonset-8jx4f\" (UID: \"e7a0137e-a621-477f-9961-17eec756c979\") " pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:19.036116 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:19.036089 5988 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:19.404533 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:19.404502 5988 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f"] Nov 03 19:51:19.408278 ip-10-0-139-161 kubenswrapper[5988]: W1103 19:51:19.408244 5988 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode7a0137e_a621_477f_9961_17eec756c979.slice/crio-c93b9d1c29ac37482633affb3f468cbbda2c9680e0ce27d8162ebc87f6f308a4 WatchSource:0}: Error finding container c93b9d1c29ac37482633affb3f468cbbda2c9680e0ce27d8162ebc87f6f308a4: Status 404 returned error can't find the container with id c93b9d1c29ac37482633affb3f468cbbda2c9680e0ce27d8162ebc87f6f308a4 Nov 03 19:51:19.428156 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:19.428126 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" event={"ID":"e7a0137e-a621-477f-9961-17eec756c979","Type":"ContainerStarted","Data":"c93b9d1c29ac37482633affb3f468cbbda2c9680e0ce27d8162ebc87f6f308a4"} Nov 03 19:51:19.985892 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:19.985861 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sclr2_71691d5d-452d-4976-998f-715fb0207c0f/limitador/0.log" Nov 03 19:51:20.434542 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:20.434505 5988 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" event={"ID":"e7a0137e-a621-477f-9961-17eec756c979","Type":"ContainerStarted","Data":"b1aed3776e8c223f533c1728f2baa9c279b9b96d7e25202e2cb3824117ea0353"} Nov 03 19:51:20.434779 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:20.434600 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:20.544350 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:20.544293 5988 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" podStartSLOduration=2.5442778649999997 podStartE2EDuration="2.544277865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:20.542970569 +0000 UTC m=+4355.424386672" watchObservedRunningTime="2026-11-03 19:51:20.544277865 +0000 UTC m=+4355.425693967" Nov 03 19:51:23.056032 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:23.055996 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_171f7ba5-c3b8-4865-82cd-783b05e3028e/alertmanager/0.log" Nov 03 19:51:23.469017 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:23.468986 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_171f7ba5-c3b8-4865-82cd-783b05e3028e/config-reloader/0.log" Nov 03 19:51:23.880820 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:23.880792 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_171f7ba5-c3b8-4865-82cd-783b05e3028e/kube-rbac-proxy-web/0.log" Nov 03 19:51:24.294010 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:24.293934 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_171f7ba5-c3b8-4865-82cd-783b05e3028e/kube-rbac-proxy/0.log" Nov 03 19:51:24.713098 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:24.713070 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_171f7ba5-c3b8-4865-82cd-783b05e3028e/kube-rbac-proxy-metric/0.log" Nov 03 19:51:25.127921 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:25.127897 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_171f7ba5-c3b8-4865-82cd-783b05e3028e/prom-label-proxy/0.log" Nov 03 19:51:25.541334 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:25.541255 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_171f7ba5-c3b8-4865-82cd-783b05e3028e/init-config-reloader/0.log" Nov 03 19:51:25.935974 ip-10-0-139-161 kubenswrapper[5988]: E1103 19:51:25.935939 5988 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p9mbd" containerName="extractor" Nov 03 19:51:26.449374 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:26.449345 5988 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tjsjt/perf-node-gather-daemonset-8jx4f" Nov 03 19:51:26.461035 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:26.461008 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5ch4r_1f3c563d-6155-43a3-bff6-5e95b9cee38d/kube-state-metrics/0.log" Nov 03 19:51:26.869638 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:26.869604 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5ch4r_1f3c563d-6155-43a3-bff6-5e95b9cee38d/kube-rbac-proxy-main/0.log" Nov 03 19:51:27.281278 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:27.281208 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5ch4r_1f3c563d-6155-43a3-bff6-5e95b9cee38d/kube-rbac-proxy-self/0.log" Nov 03 19:51:27.698534 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:27.698504 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7db59cdcd6-pqldv_74f5b2ca-aa33-444a-bd6f-56df2278d099/metrics-server/0.log" Nov 03 19:51:29.825254 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:29.825225 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vj6dq_b07cbb8d-1cfb-425b-8de7-e8ae88befa55/node-exporter/0.log" Nov 03 19:51:30.249038 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:30.248970 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vj6dq_b07cbb8d-1cfb-425b-8de7-e8ae88befa55/kube-rbac-proxy/0.log" Nov 03 19:51:30.659548 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:30.659520 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vj6dq_b07cbb8d-1cfb-425b-8de7-e8ae88befa55/init-textfile/0.log" Nov 03 19:51:32.453320 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:32.453293 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wj6cv_42604250-348d-4471-8d8a-d377e8986eaf/kube-rbac-proxy-main/0.log" Nov 03 19:51:32.865601 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:32.865570 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wj6cv_42604250-348d-4471-8d8a-d377e8986eaf/kube-rbac-proxy-self/0.log" Nov 03 19:51:33.283277 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:33.283190 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wj6cv_42604250-348d-4471-8d8a-d377e8986eaf/openshift-state-metrics/0.log" Nov 03 19:51:33.711415 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:33.711383 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56d46121-2a72-4eb8-bbd3-f548a2daac7d/prometheus/0.log" Nov 03 19:51:34.125543 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:34.125516 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56d46121-2a72-4eb8-bbd3-f548a2daac7d/config-reloader/0.log" Nov 03 19:51:34.538261 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:34.538191 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56d46121-2a72-4eb8-bbd3-f548a2daac7d/thanos-sidecar/0.log" Nov 03 19:51:34.950401 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:34.950375 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56d46121-2a72-4eb8-bbd3-f548a2daac7d/kube-rbac-proxy-web/0.log" Nov 03 19:51:35.362836 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:35.362807 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56d46121-2a72-4eb8-bbd3-f548a2daac7d/kube-rbac-proxy/0.log" Nov 03 19:51:35.774835 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:35.774764 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56d46121-2a72-4eb8-bbd3-f548a2daac7d/kube-rbac-proxy-thanos/0.log" Nov 03 19:51:36.186429 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:36.186401 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_56d46121-2a72-4eb8-bbd3-f548a2daac7d/init-config-reloader/0.log" Nov 03 19:51:36.606629 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:36.606599 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-txxgc_9b32f9a9-d324-4501-9c23-4e93b00863aa/prometheus-operator/0.log" Nov 03 19:51:37.015581 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:37.015508 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-txxgc_9b32f9a9-d324-4501-9c23-4e93b00863aa/kube-rbac-proxy/0.log" Nov 03 19:51:37.882996 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:37.882968 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74f6bf747d-rsr85_3d64f9eb-a543-4518-8d1d-0a27cac5a88a/telemeter-client/0.log" Nov 03 19:51:38.315731 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:38.315619 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74f6bf747d-rsr85_3d64f9eb-a543-4518-8d1d-0a27cac5a88a/reload/0.log" Nov 03 19:51:38.719795 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:38.719766 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74f6bf747d-rsr85_3d64f9eb-a543-4518-8d1d-0a27cac5a88a/kube-rbac-proxy/0.log" Nov 03 19:51:39.139494 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:39.139455 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c8cc96c46-k9l2m_c026965c-43dc-41e4-a73a-90ba867305a5/thanos-query/0.log" Nov 03 19:51:39.552280 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:39.552196 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c8cc96c46-k9l2m_c026965c-43dc-41e4-a73a-90ba867305a5/kube-rbac-proxy-web/0.log" Nov 03 19:51:39.964095 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:39.964048 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c8cc96c46-k9l2m_c026965c-43dc-41e4-a73a-90ba867305a5/kube-rbac-proxy/0.log" Nov 03 19:51:40.376638 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:40.376609 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c8cc96c46-k9l2m_c026965c-43dc-41e4-a73a-90ba867305a5/prom-label-proxy/0.log" Nov 03 19:51:40.788961 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:40.788877 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c8cc96c46-k9l2m_c026965c-43dc-41e4-a73a-90ba867305a5/kube-rbac-proxy-rules/0.log" Nov 03 19:51:41.201534 ip-10-0-139-161 kubenswrapper[5988]: I1103 19:51:41.201507 5988 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c8cc96c46-k9l2m_c026965c-43dc-41e4-a73a-90ba867305a5/kube-rbac-proxy-metrics/0.log"